CVP Reporting Server Troubleshooting Tips
Recently Gaman team members had a chance to upgrade UCCE environment from version 12.5 to 12.6(2). One of the elements of this upgrade was a Cisco CVP Reporting Server. This component is used to monitor and deliver reports from all CVP VXML Applications deployed on CVP VXML Servers. Normally the upgrade goes smooth but, in our case, we encountered some small problems that we were able to fix without Cisco support. Below you will find a short description of problems that we faced and brief description how we fixed the issue.
Table of Contents
DB space rootdbs is full
One of the problems that may appear is the configuration of the Informix rootdbs database. Sometimes when the CVP Reporting server must deal with large amount of data it is possible that DB will fill its space for system DB like rootdbs. This info can be found in logs – just search for the following entry:
|
X.X.X.X: kwi 24 2024 00:58:22.235 +0200: %CVP_12_6_RPT-3-INFORMIX_ALARM: [44]: DBSpace is full: 'rootdbs' -- WARNING: DBspace rootdbs is full [id:4016] |
The problem is well documented and described in Cisco documentation. Here is one of the links that describes the problem in details: Article.
In general, to fix the problem you will need to expand (add chunk space) the roodbs space. As it is Informix DB, the procedure is more complex compared to MS SQL.
In our case completing all steps in the document, fixed the issue.
Attempt to connect to database server (cvp) failed
After you start the reporting server there is a big chance that you may encounter the problems related with connection to Informix database. The problem is visible in the logs with the following sequence:
|
X.X.X.X: kwi 24 2024 00:58:11.097 +0200: %CVP_12_6_RPT-3-REPORTING_INTERNAL_ERROR: Can't initialize the JDBC Connectionjava.sql.SQLException: com.informix.asf.IfxASFException: Attempt to connect to database server (cvp) failed. [id:4003] |
In this case troubleshooting was long but at the end we understood what is happening in this case. In general, you need to assume that CVP Reporting Server needs 2 components to run.
First component is application which in this case is Call Server (windows service on CVP Reporting Server). It is responsible for gathering info, processing it and write down the data to Informix database.
Second component is Informix database – system that stores the data and publishes it to reporting systems – like CUIC.
To properly work that need to be stated in sequence – database needs to be available before application can connect to it. Having in mind this assumption we need to look how call server is configured as a Service. To gain knowledge, we can use Windows sc command runed in command line with admin privileges. Here is an output without changes:
|
C:\Cisco\CVP>sc qc CallServer [SC] QueryServiceConfig SUCCESS SERVICE_NAME TYPE START_TYPE ERROR_CONTROL BINARY_PATH_NAME LOAD_ORDER_GROUP TAG DISPLAY_NAME DEPENDENCIES SERVICE_START_NAME |
: CallServer : 10 WIN32_OWN_PROCESS : 2 AUTO_START : 1 NORMAL : C:\Cisco\CVP\CallServer\Tomcat\bin\tomcat9.exe //RS//CallServer : : 0 : Cisco CVP CallServer : Tcpip : Afd : LocalSystem |
The description of the service shows 2 dependencies: Tcpip and Afd. But this service doesn’t depend on Informix DB. What worked in our case was addition of the Informix DB dependency. But life in Cisco word isn’t simple – why? Informix DB has a very specific name cvp. To find it we needed some time.
Adding an Informix DB dependency will force Cisco Call Server service to wait a little bit longer than usual, as the Informix DB needs to report to windows that it has been properly started.
To add dependency, we can also use sc command.
|
C:\Cisco\CVP> sc config CallServer depend=Tcpip/Afd/cvp [SC] ChangeServiceConfig SUCCESS |
Once the command is issued you sc qc command (the one before change). In the DEPENDENCIES section, additional cvp should be presented.
Finalize the fix, cycle Cisco Call Server service, and check the CVP logs. The problem should disappear.
If you would like to roll back (to original configuration) maybe for Cisco TAC troubleshooting purposes, simply just issue the following command:
|
C:\Cisco\CVP> sc config CallServer depend=Tcpip/Afd |
Database uncompleted calls
One of the problems that may appear during the CVP Reporting Server start is related to uncompleted calls. What are uncompleted calls?
Uncompleted calls are calls that have been written to database with a start time but for some reason, the calls were not completed – we can say it doesn’t have completion time in the DB. Internally we call them Zombi calls.
Below is a log snippet that show how this problem is logged in CVP Reporting Server logs:
|
X.X.X.X: kwi 24 2024 07:22:52.090 +0200: %CVP_12_6_RPT-6-REPORTING_INFO: Init uncompleted call: A6FFB88000010000000094C821E611AC CallStartDate: 2024-04-23 [id:4000] X.X.X.X: kwi 24 2024 07:22:52.090 +0200: %CVP_12_6_RPT-6-REPORTING_INFO: Init uncompleted call: A830E580000100000014A1B41FE611AC CallStartDate: 2024-04-23 [id:4000] X.X.X.X: kwi 24 2024 07:22:52.090 +0200: %CVP_12_6_RPT-6-REPORTING_INFO: Init uncompleted call: A9FAA900000100000014A1B51FE611AC CallStartDate: 2024-04-23 [id:4000] X.X.X.X: kwi 24 2024 07:22:52.090 +0200: %CVP_12_6_RPT-6-REPORTING_INFO: Init uncompleted call: AE26C68000010000000094C921E611AC CallStartDate: 2024-04-23 [id:4000] X.X.X.X: kwi 24 2024 07:22:52.090 +0200: %CVP_12_6_RPT-6-REPORTING_INFO: Init uncompleted call: AF57F380000100000014A1B81FE611AC CallStartDate: 2024-04-23 [id:4000] X.X.X.X: kwi 24 2024 07:22:52.090 +0200: %CVP_12_6_RPT-6-REPORTING_INFO: Init uncompleted call: B0892080000100000014A1B91FE611AC CallStartDate: 2024-04-23 [id:4000] ....................................... X.X.X.X: kwi 24 2024 01:04:14.145 +0200: %CVP_12_6_RPT-6-REPORTING_INFO: Recover Uncompleted call: 0 CallGUID:CC2209ECF84111EE803F40F078FAFA60 Event Id: 11 CauseId: 0 [id:4000] X.X.X.X: kwi 24 2024 01:04:14.145 +0200: %CVP_12_6_RPT-6-REPORTING_INFO: Recover Uncompleted call: 0 CallGUID:CC2209ECF84111EE803F40F078FAFA60 Event Id: 11 CauseId: 0 [id:4000] X.X.X.X: kwi 24 2024 01:04:14.145 +0200: %CVP_12_6_RPT-6-REPORTING_INFO: Recover Uncompleted call: 0 CallGUID:CC2209ECF84111EE803F40F078FAFA60 Event Id: 6 CauseId: 1 [id:4000] X.X.X.X: kwi 24 2024 01:04:14.145 +0200: %CVP_12_6_RPT-6-REPORTING_INFO: Recover Uncompleted call: 0 CallGUID:CC2209ECF84111EE803F40F078FAFA60 Event Id: 4 CauseId: 0 [id:4000] |
The environment that we were working on had multiple entries with Zombi calls that in the end, stopped initialization process of the CVP Reporting Server and didn’t allow it to start.
Before we move forward, we tried to understand what was causing the issue. After doing some tests we found out that the problem is related to the procedure how you close CVP Reporting Sever. If you have planned maintenance, it is recommended to use graceful shutdown. But this doesn’t cover everything. In case when the CVP Reporting Server will be closed suddenly – someone will force shutdown virtual machine from VMWare – there is a big chance that calls processed by CVP will stay in uncompleted state.
This is why the below procedure can be used after such situations.
So how to fix the problem?
CVP Reporting Server is installed with tool that may be used to access the database. The tool is called DbVisualiser. We have used is as it is a part of CVP, but if you use some different tool, it also can be used. Below screen presents the tool search from Windows Start button:
Once the tool is started, you will need to fill in the DB connection details. In our case we used the admin account. Here are connection details:
Database section:
- Database Server – enter IP address of the CVP Reporting server.
- Database Port – always: 1526
- Database Serve (second on list) – always: cvp.
Authorization section:
- Database Userid – if you connect as administrator, enter: cvp_dbadmin.
- Database Password – password assigned to entered user account. If you are using admin account, the DB admin password
can be set on using this procedure:
link
Here is a screenshot of the database connection form with database details:
Once connection is established, please open a query window as we will make some queries to the database.
To find out the number of Zombi calls (calls that don’t have end date), please execute the following query:
|
SELECT * FROM call WHERE enddatetime IS NULL |
If the result shows more then 0, then the calls require some fixing. The fix is very simple and assumes that the end date will have the value of time when the fix be applied. The query in this can look like:
|
UPDATE call SET enddatetime=CURRENT WHERE enddatetime IS NULL |
Number of rows updated should match the number of rows identified for fix.
Please remember that if you have a Zombi calls the that were written long time ago (ex. week or so) the above query may impact the calculation of the call duration. In such cases the better assumption would be to set duration of this calls to 0. To do that you may use the below query:
|
UPDATE call SET enddatetime=startdatetime WHERE enddatetime IS NULL |
You may also use your own query. All is up to you.