Post new topic Reply to topic  [ 7 posts ] 

Board index : TeleFlow Forums : TeleFlow Server & Monitor

Author Message
 Post subject: Heavy disk & CPU activity slooooows down TFServer
PostPosted: Tue Jun 17, 2008 1:00 pm 
Offline
User avatar

Joined: Fri Feb 06, 2004 11:05 am
Posts: 115
Location: Costa Rica
I have a server with 46 channels from two T1s. As both T1s are provider by different telephony companies, the idea is to use them this way:

  • 23 channels from trunk 0 for incoming calls (VOICEPOOL)
  • 20 channels from trunk 1 for call transfering to a customer center(TRANSFERPOOL)
  • 3 channels from trunk 1 for fax output (FAXPOOL)

If a customer being attended in any channel from the VOICEPOOL wants to be attended by a customer-care agent, the IVR application will ask the apps in the TRANSFERPOOL to initiate an outbound call to the trunk 1's pilot number. Once there is an answer, the channels get switched and the application will remain in background waiting for the hangup event from any of the involved channels.

The quantity of channels in the TRANSFERPOOL is just a negative assumption (the worst case scenario), in which that many people may engage in customer-care attention at the same time... Yes, once in production that number may get lower.

I followed the TeleFlow demos to implement that. However, once it's running, the TRANSFERPOOL apps get to consume the CPU entirely... they search for "outbound call requests" (files on disk) every second or so and being 20 of them it gets quite busy. This slow-down also impacts the performance of the applications from the VOICEPOOL, making them run much slower than when they run in the Simulator.

I guess that making them to search every 5 seconds, the CPU load will decrease, but this may impact the time the customer has to wait before getting some attention.

Do you have a suggestion (other than using ODBC or SQL) to improve this performance without bogging down the server? Is there a way to 'serialize' the search, like to make every app in the TRANSFERPOOL to search every 21 seconds and then making app in channel 24 to sleep after its search, then app in channel 25 to search and sleep in the next second, then app in channel 26 search and sleep in the next one and so on?

Thanks in advance.


Back to top
 Profile  
 
 Post subject: Re: Heavy disk & CPU activity slooooows down TFServer
PostPosted: Tue Jun 17, 2008 3:32 pm 
Offline
Site Admin

Joined: Wed Dec 31, 1969 5:00 pm
Posts: 329
Location: Vancouver, BC
Go ahead and increase the wait time between checks on the TRANSFERPOOL applications. Your callers will not notice. If memory serves, those demos include a simulated ring-back in the in-bound applications. This means that while the TRANSFERPOOL is waiting, your callers will think they are being transferred. I would recommend a 6-second wait between checks, as this is one ring-cadence cycle. This will guarantee that your caller's wait time will increase by no more than one ring.

The next thing I would do is to build indexes on the table or tables that are queried by TRANSFERPOOL, to ensure the queries are as quick as they can possibly be. If you are using the demo table unchanged, "swipool.dbf" has no indexes on it at all. I recommend one index on the "Port" field and another on the "Status" field.

These two changes alone will probably fix the problem.

If it doesn't, here's some thoughts off the top of my head:

'Serializing' the search is a good idea. I'm not really sure how you might accomplish that. You could add a field (or fields) to the "switch pool" table so that an out-bound port could tell if it was the "top of the list". If so, it would set its wait time to 2 seconds, and if not, set it to 20 seconds. That's a simplified version of what you want to do.

Consider using the UDP steps. You could have each TRANSFERPOOL bind to an distinct IP port at startup, using the "UDP Bind" step. Instead of the Wait step, have the port listen with the "UDP Receive" step. When an in-bound port decides to request a transfer, instead of writing to your "switch pool" table, send a UDP message, via "UDP Send" step to the TRANSFERPOOL port. Does this makes sense?

Note that you probably won't be able to get rid of the database interaction all together. Your in-bound ports need to be able to tell when an outbound port is available to place a call. You could still use the "switch pool" table to do that. A TRANSFERPOOL port would perform an update on that table every time it changes state (from idle, to on a call, and vice versa). The in-bound apps would query the "switch pool" table to find a free port. You'll need to be careful how you construct this, as it would be possible for two in-bound ports to select the same out-bound port.

You might be able to construct something even fancier using TCP (also available in TeleFlow steps). In this case, I'd make a "transfer manager" application, that could handle all the requests to guarantee no two in-bound ports request the same outbound port. The signals from the in-bound ports, to the manager would be more like "I need a port to transfer my caller", and the manager would choose and signal the out-bound port.

This reminds me of a pet-project I've always wanted to do, but never have had the time. I'd love it if TeleFlow Server itself could handle these inter-port co-ordinations. Essentially, I want to "internalize" the switching pool apps so they are an inherent, native, capability of TFServer. Sadly, this will not happen in time to help your project.

Honestly, as much fun as it is to contemplate these systems, the simple version is probably best. I'd stick with my first two suggestions, and see what happens.


Back to top
 Profile WWW 
 
 Post subject: Re: Heavy disk & CPU activity slooooows down TFServer
PostPosted: Tue Jun 17, 2008 5:07 pm 
Offline
Site Admin

Joined: Wed Dec 31, 1969 5:00 pm
Posts: 329
Location: Vancouver, BC
One other thing...
Quote:
As both T1s are provider by different telephony companies, the idea is to use them this way:

You might have problems making this work. Your NMS configuration will require that you get clocking information from one of these two circuits, but not both. you may find you have communication errors between the 2nd T1 span on the card and the other telephone company.


Back to top
 Profile WWW 
 
 Post subject: Re: Heavy disk & CPU activity slooooows down TFServer
PostPosted: Wed Jun 18, 2008 11:55 am 
Offline
User avatar

Joined: Fri Feb 06, 2004 11:05 am
Posts: 115
Location: Costa Rica
I will later comment on your suggestions for improving the communication and performance between the VOICEPOOL & TRANSFERPOOL applications.

For now I want to ask about your last comment: NMS Support suggested me to use the STANDALONE clockmode with the internal OSCillator as the clock source, because I'm using a single CG6060 board (they insist that clocking propagation is used with two or more boards).

So, if I get two T1s from different providers into the same board that generates its own clock, would the communication errors still happen when switching channels from the first T1 to the second one?

What would happen if I ditch the second provider (Codetel) and use two T1s from the same provider (Tricom) instead? May I expect no communication errors in this scenario? Should I still use some other clocking configuration or can I continue using the internal oscillator?

Finally, Tricom's T1 is a PRI(ISDN) using NI2 as the protocol variant. This is coming from the PSTN, so it's not connected to any PBX (but it has a hunt-group/pilot number managed by Tricom). Is it possible to define a pool of 20 channels for incoming calls, another pool of 3 channels for outbound calls and perform a 2B-Transfer between both pools (a shorter transfer method to the call center agents in another PSTN number)?


Back to top
 Profile  
 
 Post subject: Re: Heavy disk & CPU activity slooooows down TFServer
PostPosted: Wed Jun 18, 2008 12:39 pm 
Offline
User avatar

Joined: Fri Feb 06, 2004 11:05 am
Posts: 115
Location: Costa Rica
Tim wrote:
Go ahead and increase the wait time between checks on the TRANSFERPOOL applications. Your callers will not notice.


Ok, I'll test that.

Quote:
The next thing I would do is to build indexes on the table or tables that are queried by TRANSFERPOOL, to ensure the queries are as quick as they can possibly be.


Actually I'm not using tables, just disk files. I guess the disk overhead is also affecting considerably...

Quote:
'Serializing' the search is a good idea. I'm not really sure how you might accomplish that.


I was thinking: TRANSFERPOOL applications run on channels 24 to 43 and each one knows its port number (@PORT). What if I make each application to check once and then wait for (@PORT-20) seconds? That would serialize them: Each second, except for the first 4 (nothing critical), there will be a single app checking for transfer request files on disk. This minimizes disk overhead, optimizes CPU performance and minimizes the chance that two or more apps from that pool will find the same file and try to perform the call and ask for the switching simultaneoulsy with other apps.

This might work for awhile, but once the switch is over (any party hangs up), the TRANSFERPOOL application may get de-synchronized with the others, given that the channels were release at a random time, after which the app continues waiting and checking for transfer requests every 20 seconds... Gets tricky!

Quote:
Consider using the UDP steps. You could have each TRANSFERPOOL bind to an distinct IP port at startup, using the "UDP Bind" step. Instead of the Wait step, have the port listen with the "UDP Receive" step. When an in-bound port decides to request a transfer, instead of writing to your "switch pool" table, send a UDP message, via "UDP Send" step to the TRANSFERPOOL port. Does this makes sense?


If every TRANSFERPOOL app binds to a distinct port, there wouldn't be a single TRANFERPOOL port to send requests with "UDP Send"...

There should be some kind of 'Transfer Requests Manager' which receives the requests from the VOICEPOOL apps sent to a single port and channels them to a managed list of available/busy applications in the TRANSFERPOOL.

UDP steps here sound like a one-way communication however...

Quote:
Note that you probably won't be able to get rid of the database interaction all together. Your in-bound ports need to be able to tell when an outbound port is available to place a call.


Right, I need two-way communications here.

Quote:
You might be able to construct something even fancier using TCP (also available in TeleFlow steps). In this case, I'd make a "transfer manager" application, that could handle all the requests to guarantee no two in-bound ports request the same outbound port. The signals from the in-bound ports, to the manager would be more like "I need a port to transfer my caller", and the manager would choose and signal the out-bound port.


Oh boy 8) Well, guess that thinking alike is common behaviour between seasoned developers, eh?

Problem is I'm already certifying the IVR application for the customer, so I don't have more time right now to develop a transfer manager application :x

Quote:
This reminds me of a pet-project I've always wanted to do, but never have had the time. I'd love it if TeleFlow Server itself could handle these inter-port co-ordinations. Essentially, I want to "internalize" the switching pool apps so they are an inherent, native, capability of TFServer. Sadly, this will not happen in time to help your project.

Honestly, as much fun as it is to contemplate these systems, the simple version is probably best. I'd stick with my first two suggestions, and see what happens.


Alright, I'll see what comes up and document results here.

Thanks a lot!


Back to top
 Profile  
 
 Post subject: Re: Heavy disk & CPU activity slooooows down TFServer
PostPosted: Wed Jun 18, 2008 6:10 pm 
Offline
Site Admin

Joined: Wed Dec 31, 1969 5:00 pm
Posts: 329
Location: Vancouver, BC
fesquivel wrote:
For now I want to ask about your last comment: NMS Support suggested me to use the STANDALONE clockmode with the internal OSCillator as the clock source, because I'm using a single CG6060 board (they insist that clocking propagation is used with two or more boards).

So, if I get two T1s from different providers into the same board that generates its own clock, would the communication errors still happen when switching channels from the first T1 to the second one?

What would happen if I ditch the second provider (Codetel) and use two T1s from the same provider (Tricom) instead? May I expect no communication errors in this scenario? Should I still use some other clocking configuration or can I continue using the internal oscillator?

Finally, Tricom's T1 is a PRI(ISDN) using NI2 as the protocol variant. This is coming from the PSTN, so it's not connected to any PBX (but it has a hunt-group/pilot number managed by Tricom). Is it possible to define a pool of 20 channels for incoming calls, another pool of 3 channels for outbound calls and perform a 2B-Transfer between both pools (a shorter transfer method to the call center agents in another PSTN number)?

Firstly, you should consider NMS's support over mine. I'm not as experienced in their products as they are!

My experience has been that if you use the on-board oscillator to drive the board's clock that you get occassional frame-slips in the communication between your board and the telco. I don't know what this actually does to your application. It may result in dropped calls, or it may not be noticeable at all. The only way you'd be able to tell for sure is to try it and see.

If you get both your T1's from the same service provider, you will not have a problem synchronizing your clock with theirs. If you do, it is the fault of your service provider, and their responsibility to fix. In this case, it is best to set your CG board to get its clock from one of the two T1's. I usually go with #1. The lines in your board-level config file should look something like this:
Code:
Clocking.HBus.ClockSource            = NETWORK
Clocking.HBus.ClockSourceNetwork     = 1

With respect to your 2B-Transfer question, the answer is "Yes!" That is an ideal scenario for when you don't want to manage the switched call between the two parties (examples of managing the call would be timing its duration, being able to continue to provide IVR functionality after one of the parties hangs up, etc). TeleFlow has a Step to request that the network perform a 2B-Transfer. The step is "Transfer Step". There should be an explanation in the help on how to use it.

The most important things to know are (1) you have to get the system call reference from the "other" port in order to tell the telco's switch which B-channel you want to be transferred to, and (2) a successful 2B-Transfer results in a hang-up on both ports. The following link will download a revision of our "Transfer Demo 2" sample application that uses the Transfer Step instead:
2B Transfer Demo


Back to top
 Profile WWW 
 
 Post subject: Re: Heavy disk & CPU activity slooooows down TFServer
PostPosted: Wed Jun 18, 2008 6:15 pm 
Offline
Site Admin

Joined: Wed Dec 31, 1969 5:00 pm
Posts: 329
Location: Vancouver, BC
Quote:
Problem is I'm already certifying the IVR application for the customer, so I don't have more time right now to develop a transfer manager application
Exactly why I think increasing your wait times will be your best option.

Another thing to consider is how many files are in the folder where these applications are looking for flat files? Windows folder access gets progressively slower the more entries you have in a single folder (directory in the old lingo). Instead of building an index, you could ensure you had one folder dedicated to transfer communications and always delete files when you are finished with them.

Good luck, and let us all know what happens!


Back to top
 Profile WWW 
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 7 posts ] 

Board index : TeleFlow Forums : TeleFlow Server & Monitor


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Style by Midnight Phoenix & N.Design Studio
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.