Weird TCP error when invoking a webservice

fesquivel · **Posted:** Wed Sep 01, 2010 1:19 pm

I'm recently testing new logging features and discovered a weird behaviour when invoking a particular webservice:

Code:

  [384] HTTP
    Silence TIMEOUT '@GBL_WEBSERVICETIMEOUT' evaluates to '10000'
    HTTP Initial Connection TIMEOUT '@GBL_WEBSERVICETIMEOUT' evaluates to '10000'
    HTTP Initial Connection Retries is ''
    Empty HTTP Initial Connection Retries defaults to '2'
    Script code is 'wait'
    Script code 'wait' corresponds to file 'Vox\Spn\Scripts\Wait.wav'
    URL '@vSiteCall' evaluates to <ommited: sensitive data>
    HTTP Request Message Body is ''
    Setting local variable '@VXMLLOAN' to ''
    Playing 'Wait.wav'
    Connection attempt 1
    Connection attempt 2
    ERROR: Unable to connect to host
       'WSAENOTSOCK'
    Setting local variable '@VXMLLOAN' to ''
    FAILURE

Hiding some sensitive data with 'X', the contents of @vSiteCall are:

Code:

http://webservicehbexecuter/Executer.asmx/Execute?pIdentifier=PRLoanInformation&pValues=6XXX1~00XXXXXX909~IVRID~WB~WBP~2XXX1~ES&pClient=

The untouched contents work just fine when pasted in IE8 in the same computer: I get all the XML data I need to parse for later dictation. But the IVR application just fails with that WSAENOTSOCK error.

According to Visual Studio's Help, that means:

Quote:

Socket operation on nonsocket. An operation was attempted on something that is not a socket. Either the socket handle parameter did not reference a valid socket, or for the select() function, a member of an fd_set structure was not valid.

So, what's wrong? The IVR app can invoke the same webservice with different parameters (other procedures identified in pIdentifier) just fine, what's wrong with this particular call?

fesquivel · **Posted:** Fri Sep 03, 2010 9:07 am

Just an update: For that test I ran the Simulator in my development PC (as always), so I installed a copy of the sources being debugged on a different directory on the Production server (where the IIS and our webservices reside as well).

I then ran the Simulator using this copy and the webservice invocation worked just fine... So the same URL worked locally, just not remotely from the Simulator on my PC (although it worked fined from my IE8).

I'm not sure if this is a TeleFlow issue, an IIS issue, an issue in the webservice itself, or some kind of remote access issue (network permissions or filters of some kind), but as I'm truely intrigued, I'll test again using a network sniffer, in order to capture the TCP handshake and see what's the origin of that WSAENOTSOCK error. I'll be posting any findings...

Chris · **Posted:** Fri Sep 03, 2010 9:45 am

Please do! I was about to respond to your post, but I actually have little to suggest at this stage, and now that you have found it working in one location and not another, it's even more a mystery.

Having said that...

Are you using the same version of TeleFlow on the development box it isn't working on, from that on the production system?

I was also going to suggest that maybe the tilde(~) character is causing some unpredictable and/or inconsistent behavior. I don't actually think this is the case, or that if this was they problem you would get a WSAENOTSOCK error, but it is one thing that sticks out.

The TeleFlow HTTP step doesn't encode your URL for you, as does a browser. This means that if your URL contains "illegal" URL characters, you will need to encode them yourself. Tilde isn't illegal, but was pretty close to it in the past, and it is still recommended (though not always necessary) that it be properly encoded instead of using the character itself directly in the URL.

If you wanted to pursue this anyway, you could build a URLEncode flowchart, where you pass in the URL you have now, and receive back the encoded URL. Inside this flowchart, you can have a flowchart that takes the string and replaces all tilde(~) characters with their appropriate encoded value. (For tildes, the replacement string is %7e) There is a template flowchart that does string replacement called FindAndReplaceString. To add a template flowchart to your application, go to the TAP screen, flip to the "Templates" tab of the Steps Toolbox, click FindAndReplaceString, and drop it onto the TAP screen flowchart space.

If you have other URLs and find you need to account for additional invalid characters, you can then add those to the URLEncode flowchart, until you cover all you need.

Let us know what you uncover with the packet sniffer. Thanks!

fesquivel · **Posted:** Fri Sep 03, 2010 12:26 pm

I just ran the same test with the sniffer and will submit the Wireshark's CAP file to support@engenic.com for your revision.

This is what I found:

Packets 1 .. 3: Initial handshake for HTTP query #1 (user authentication)
Packet 4: HTTP GET command that sends the userID and password
Packet 5: HTTP XML response, with RESPONSECODE=000 (means OK)
Packets 6 .. 8: Final handshake to close query #1

That's an example that shows the webservice working just fine. Query #2, which gets the customer's product list, runs exactly the same (initial handshake, GET command, XML response and final handshake) from packets 9 to 19.

Then starts the failing query #3:

Packet 20: It's the app's initial handshake packet (SYN)
Packet 21: It's the IIS's acknowledgement accepting the connection
Packets 22 and 23: These are repeated acknowledgments given that the application did not answer back to finish the handshake
Packets 24 and 25: These seem to be two more connection attempts from the application to the remote server (as if the app did not see the previous acknowledgments!), but they are issued to a different remote server at 75.101.141.136. Why!?

So, the misbehaviour seems to start at the very beginning of the connection attempt but only when I try to issue the URL call I showed in my first post starting this topic.

I'm going to do a full packet capture to see if there is some DNS/UDP name resolve in this third query that yields that strange IP address.

fesquivel · **Posted:** Fri Sep 03, 2010 3:30 pm

I forgot to mention I'm using the same Designer & Simulator versions in both computers: v2009.6.5.1255 and build 2010.01.14, respectively.

This time I couldn't see the connection attempts to the strange IP address, but there weren't any other type of communication between the hosts (my PC and the IVR server which hosts the webservices).

I just send an extract from the log file (the HTTP invocations) as well as the CAP file from Wireshark.

Any hints?

Chris · **Posted:** Fri Sep 03, 2010 4:18 pm

Let's try getting a look at the results in both the capture and the TeleFlow log if you change your http step so it has:

"HTTP Initial Connection Retries" set to 0.

I'm concerned that the 2 retries - the default if you leave the connection retries blank - might be confusing the issue a little. In particular, it shows
Connection Attempt 1
Connection attempt 2
in VERY rapid succession. There might be something to this. It doesn't show what the error for the first of these connection failures is, so the 'WSAENOTSOCK' might be misleading/not the real error.

I'm not seeing anything particularly revealing in the captures so far, other than the obvious fact that the connection that should be proceeding obviously isn't. (which we already know from the TeleFlow side and the socket error)

When this issue is resolved, you can of course put "HTTP Initial Connection Retries" to whatever you wish for production purposes.

fesquivel · **Posted:** Wed Sep 08, 2010 10:06 am

Well, I guess it's another thing...

Code:

53:35.905:  [384] HTTP
53:35.905:     Silence TIMEOUT '@GBL_WEBSERVICETIMEOUT' evaluates to '10000'
53:35.905:     HTTP Initial Connection TIMEOUT '@GBL_WEBSERVICETIMEOUT' evaluates to '10000'
53:35.905:     HTTP Initial Connection Retries is '10'
53:35.905:     Script code is 'wait'
53:35.905:     Script code 'wait' corresponds to file 'Vox\Spn\Scripts\Wait.wav'
53:35.920:     URL '@vSiteCall' evaluates to <...>
53:35.920:     HTTP Request Message Body is ''
53:35.920:     Setting local variable '@VXMLLOAN' to ''
53:35.920:     Playing 'W:\...\Vox\Spn\Scripts\Wait.wav'
53:35.936:     Connection attempt 1
53:35.936:     Connection attempt 2
53:35.936:     ERROR: Unable to connect to host
53:35.936:        'WSAENOTSOCK'
53:38.842:     Setting local variable '@VXMLLOAN' to ''
53:38.842:     FAILURE

Again, the @vSiteCall variable is [url]'http://webservicehbexecuter/Executer.asmx/Execute?pIdentifier=PRLoanInformation&pValues=54386~00101535649~IVRID~WB~WBP~22369~ES&pClient='[/url], which is the same used for Login or querying other products' base information when pIdentifier gets other values...

That's the weird thing: It's the same webservice (Executer.asmx) for all purposes, but this odd behaviour only happens when its first parameter, pIdentifier, gets the value "PRLoanInformation". How's that?

The sniffer shows exactly the same behaviour: 1 SYN packet from my PC to the server, 3 SYN+ACK packets from the server and that's it, no more traffic... the handshake doesn't finish, so the connection is not established at all, unlike in the first steps (login query & product list query) where the 3-step handshake is completed.

Chris · **Posted:** Wed Sep 08, 2010 2:42 pm

It looks like you bumped your "HTTP Initial Connection Retries" up from empty (which gets set to 2) to 10.

We really want to see what the result is when it is set to 0.

It might be the same, but if the error with no connection retries is different, this might provide another (or a better) clue as to what is amiss.

Can you try once more with the value set to 0? Thanks!

fesquivel · **Posted:** Fri Sep 10, 2010 10:19 am

Oh, boy... I'm so sorry: I did not pay attention to your previous request of setting the "HTTP Initial Connection Retries" to 0 and just assumed you wanted me to try more connections attempts (that's why I used 10).

Anyway, here are the results with 0 in two different invocations of the same webservice, but using different functions (and some editing for legibility):

Code:

11:02:59.138:     [87] HTTP
11:02:59.497:        Silence TIMEOUT '@GBL_WEBSERVICETIMEOUT' evaluates to '10000'
11:02:59.497:        HTTP Initial Connection TIMEOUT '@GBL_WEBSERVICETIMEOUT' evaluates to '10000'
11:02:59.497:        HTTP Initial Connection Retries is '0'
11:02:59.497:        Script code is 'wait'
11:02:59.497:        Script code 'wait' corresponds to file 'Vox\Spn\Scripts\Wait.wav'
11:02:59.497:        URL '@SiteCall' evaluates to 
                         'http://webservicehbexecuter/Executer.asmx/Execute?pIdentifier=ADLoginIVR&pValues=00101535649~IVRID~WB~ES~1197~&pClient='
11:02:59.497:        HTTP Request Message Body is ''
11:02:59.497:        Setting global variable '@XML_AUTH' to ''
11:02:59.497:        Playing 'W:\Projects\Systems\IVR\PDOM\Application\IVR_APP\SourceCode\Vox\Spn\Scripts\Wait.wav'
11:02:59.528:        Connection attempt 1
11:02:59.731:           numWaitObjects = 5, WAIT_OBJECT_0 + 3 = 3, ret = 3  - @@@
11:02:59.731:        Connection successful
11:02:59.731:        Sending request: 
               'GET /Executer.asmx/Execute?pIdentifier=ADLoginIVR&pValues=00101535649~IVRID~WB~ES~1197~&pClient= HTTP/1.0
               User-Agent: TeleFlow
               Accept: */*
               Host: webservicehbexecuter:80
               '
11:02:59.731:        Request successful, waiting for response
11:03:00.122:           numWaitObjects = 5, WAIT_OBJECT_0 + 3 = 3, ret = 3  - @@@
11:03:00.138:        Response received
11:03:02.434:        Setting global variable '@XML_AUTH' to '<?xml version="1.0" encoding="utf-8"?>
               <RESPONSE>
                 <RESPONSECODE>000</RESPONSECODE>
                 <RESPONSETEXT>EJECUCION EXITOSA</RESPONSETEXT>
                 <ROWCOUNT>1</ROWCOUNT>
                 <ROW1>
                    <mostrar_mensaje />
                    <client_id>54386</client_id>
                    <delegate_id>00101535649</delegate_id>
                    <id_type>IVRID</id_type>
                    <sys_code>WB</sys_code>
                    <profile_code>WBP</profile_code>
                    <pwd_change>N</pwd_change>
                    <muestra_datos>N</muestra_datos>
                    <muestra_contrato>N</muestra_contrato>
                    <id_kind>ID</id_kind>
                  </ROW1>
                  <TIME>ABRIR ARCHIVO DE CONFIGURACION - Duracion: 0 segs 0 milisegs
               BUSCA EL METODO - Duracion: 0 segs 0 milisegs
               SETEAR LOS VALORES - Duracion: 0 segs 0 milisegs
               CREA LOS PARAMETROS ORACLE - Duracion: 0 segs 0 milisegs
               EJECUTANDO PAQUETE - Duracion: 0 segs 171 milisegs
               BUSCANDO SB_RESULT - Duracion: 0 segs 31 milisegs
               CREA EXECUTION RESULT - Duracion: 0 segs 0 milisegs
               GENERA XMLRESPONSE - Duracion: 0 segs 0 milisegs
               </TIME>
               </RESPONSE>'

Here is the later invocation, same webservice, but with the failing pIdentifier:

Code:

03:07.809:     [384] HTTP
03:07.809:        Silence TIMEOUT '@GBL_WEBSERVICETIMEOUT' evaluates to '10000'
03:07.809:        HTTP Initial Connection TIMEOUT '@GBL_WEBSERVICETIMEOUT' evaluates to '10000'
03:07.809:        HTTP Initial Connection Retries is '0'
03:07.809:        Script code is 'wait'
03:07.809:        Script code 'wait' corresponds to file 'Vox\Spn\Scripts\Wait.wav'
03:07.809:        URL '@vSiteCall' evaluates to 
                     'http://webservicehbexecuter/Executer.asmx/Execute?pIdentifier=PRLoanInformation&pValues=54386~00101535649~IVRID~WB~WBP~22369~ES&pClient='
03:07.809:        HTTP Request Message Body is ''
03:07.809:        Setting local variable '@VXMLLOAN' to ''
03:07.809:        Playing 
                     'W:\Projects\Systems\IVR\PDOM\Application\IVR_APP\SourceCode\Vox\Spn\Scripts\Wait.wav'
03:07.825:        Connection attempt 1
03:07.825:        Connection attempt 2
03:07.825:        ERROR: Unable to connect to host
03:07.825:           'WSAENOTSOCK'
03:10.747:        Setting local variable '@VXMLLOAN' to ''
03:10.747:        FAILURE

I will send the entire log and sniffer captures to support@engenic.com for your revision.

Chris · **Posted:** Mon Sep 13, 2010 12:32 pm

No worries about the connection attempts. As it turns out, it still tried a second connection anyway (fortunately, that tells us a little about what is happening).

We don't suspect this is a network related issue at this point; the WSAENOTSOCK is probably a misleading error based on the way the second connection attempt occurs.

Can you try in your test environment going straight from Wait for Call to the http step that currently fails, and having its success path go to a second copy of the same step with the same url/parameters/etc?

Please provide a log of this test. We can do without the wire capture until/unless we get back to suspecting a networking issue.

Thanks!

fesquivel · **Posted:** Thu Oct 07, 2010 10:24 am

Sorry for the delay... real work sometimes gets in the way of coding/testing for fun.

I just did what you asked and here are the log extracts... I copied the XML step from the flowchart where it's failing and put it after the Wait for Call step, as well as the initializers for the two variables in use. And it worked right away...

Code:

06:48.783: [220] HTTP
06:49.111:    Silence TIMEOUT '@GBL_WEBSERVICETIMEOUT' evaluates to '10000'
06:49.111:    HTTP Initial Connection TIMEOUT '@GBL_WEBSERVICETIMEOUT' evaluates to '10000'
06:49.111:    HTTP Initial Connection Retries is ''
06:49.111:    Empty HTTP Initial Connection Retries defaults to '2'
06:49.111:    Script code is 'wait'
06:49.111:    Script code 'wait' corresponds to file 'Vox\Spn\Scripts\Wait.wav'
06:49.111:    URL '@vSiteCall' evaluates to 'http://webservicehbexecuter/Executer.asmx/Execute?pIdentifier=PRLoanInformation&pValues=54386~00101535649~IVRID~WB~WBP~22369~ES&pClient='
06:49.111:    HTTP Request Message Body is ''
06:49.111:    Setting local variable '@VXMLLOAN' to ''
06:49.111:    Playing 'W:\Projects\Systems\IVR\PDOM\Application\IVR_APP\SourceCode\Vox\Spn\Scripts\Wait.wav'
06:49.127:    Connection attempt 1
06:49.314:       numWaitObjects = 5, WAIT_OBJECT_0 + 3 = 3, ret = 3  - @@@
06:49.314:    Connection successful

I then ran the application the normal way (logging in, selecting the account from the main menu) and it failed just as always...

Code:

10:52.200: [384] HTTP
10:52.216:    Silence TIMEOUT '@GBL_WEBSERVICETIMEOUT' evaluates to '10000'
10:52.216:    HTTP Initial Connection TIMEOUT '@GBL_WEBSERVICETIMEOUT' evaluates to '10000'
10:52.216:    HTTP Initial Connection Retries is ''
10:52.216:    Empty HTTP Initial Connection Retries defaults to '2'
10:52.216:    Script code is 'wait'
10:52.216:    Script code 'wait' corresponds to file 'Vox\Spn\Scripts\Wait.wav'
10:52.216:    URL '@vSiteCall' evaluates to 'http://webservicehbexecuter/Executer.asmx/Execute?pIdentifier=PRLoanInformation&pValues=54386~00101535649~IVRID~WB~WBP~22369~ES&pClient='
10:52.216:    HTTP Request Message Body is ''
10:52.216:    Setting local variable '@VXMLLOAN' to ''
10:52.216:    Playing 'W:\Projects\Systems\IVR\PDOM\Application\IVR_APP\SourceCode\Vox\Spn\Scripts\Wait.wav'
10:52.232:    Connection attempt 1
10:52.232:    Connection attempt 2
10:52.232:    ERROR: Unable to connect to host
10:52.232:       'WSAENOTSOCK'
10:55.138:    Setting local variable '@VXMLLOAN' to ''
10:55.138:    FAILURE

I'll send the full logs to the usual e-mail.

Chris · **Posted:** Wed Oct 13, 2010 9:22 am

Reviewing your last set of logs, I noticed that in the log where the http step fails, there was previously some touch tone input from the caller.

I tried some testing here, and found that if I had a Get TT String ahead of the http step, and pressed more keys than that Get TT String will accept, the http step failed like yours does. This may not be exactly what is happening to you, but the behavior in the step is the same. This issue should affect only the Simulator.

We will address this issue and improve the logging so we can see what is interrupting the normal flow of the step in a future release of TeleFlow.

Could you also some additional tests and provide logs from these? (each number below is one change and test we need a log of)

1) Put a 500ms interruptible wait just ahead of the failing http step.

2) Leaving the wait from #1 above (after testing and logging the results), add a Clear TT step just after the wait.

3) Leave the wait in the flow, but move the Clear TT step to just before the wait.

4) Eliminate the new steps, remove the audio file from the http step, and put it in a Play step just prior to the http step.

Thanks!

fesquivel · **Posted:** Fri Jul 06, 2012 3:38 pm

I'm sorry to answer back almost two years later on this issue, but I was able to test your suggestion up until today...

I will test [2] and [3] in a few minutes, but just [1] worked for me: The HTTP Step after the interruptible Wait of 500ms performs the call and gets the answer back just fine. I'll see if a 50ms delay works as well, just to save 450ms...

fesquivel · **Posted:** Fri Jul 06, 2012 3:58 pm

Alright: Suggestions [1] (with 50ms instead of 500ms), [2], [3] and [4] all worked right away.

I'm sticking with solution [4] to avoid the Wait altogether.

Thanks a lot!

Chris · **Posted:** Tue Jul 10, 2012 9:53 am

Thank you for letting us know you were able to resolve the issue!

We've logged the Simulator bug for a future fix.