Page 1 of 2 12 LastLast
Results 1 to 10 of 16
  1. #1
    Power Member
    Join Date
    Sep 2013
    Location
    Coimbatore
    Posts
    121

    Lightstreamer getting issue on connection get grows

    Hello,
    We are facing the issue like connection getting disconnect on particular time(We are facing this issue on every day 6.30 PM exactly Indian time). We are using the Lightstreamer server with java remote adapter. Everything is working fine and client getting response from LS server. Adapter also working fine and sending data to LS server.

    But for everyday 6.30 PM we are facing the issue like connection getting too over, we monitor the dashboard. Clients connection are getting close and reconnect or reopen the connection. LS server getting not responsible getting struggle to handle the connection.

    I have attached some logs in this,
    Suddenly got increased the connection and LS not respond to clients. I have faced some error on morning and also while facing the issue , I have below attached the errors from dashboard and screenshots. For Indian time 6.30 daily facing this issue. How to find whether my LS adapter code having issue or some other things getting issue.


    13:47:06,260SERVER POOLED THREAD 7Unexpected error in notifyUser: Connection closed while facing request 620000016415aec77e:
    13:47:06,260SERVER POOLED THREAD 7Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request 620000016415aec77e.
    13:47:06,354SERVER POOLED THREAD 28Unexpected error in notifyUser: Connection closed while facing request 630000016415aec77e:
    13:47:06,354SERVER POOLED THREAD 28Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request 630000016415aec77e.
    13:47:11,783SERVER POOLED THREAD 31Unexpected error in notifyUser: Connection closed while facing request 640000016415aec77e:
    13:47:11,783SERVER POOLED THREAD 31Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request 640000016415aec77e.


    18:35:18,625SERVER POOLED THREAD 19Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request f9fba0000016415b1fc70.
    18:35:18,625SERVER POOLED THREAD 29Unexpected error in notifyUser: Connection closed while facing request f9fbb0000016415b1fc70:
    18:35:18,625SERVER POOLED THREAD 29Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request f9fbb0000016415b1fc70.
    18:35:18,625SERVER POOLED THREAD 19Unexpected error in notifyUser: Connection closed while facing request f9fbc0000016415b1fc70:
    18:35:18,625SERVER POOLED THREAD 19Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request f9fbc0000016415b1fc70.
    18:35:18,641SERVER POOLED THREAD 19Unexpected error in notifyUser: Connection closed while facing request f9fbd0000016415b1fc70:
    18:35:18,859SERVER POOLED THREAD 25Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request f9fd50000016415b1fc70.
    18:35:18,859SERVER POOLED THREAD 5Unexpected error in notifyUser: Connection closed while facing request f9fd60000016415b1fc70:
    18:35:18,859SERVER POOLED THREAD 5Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request f9fd60000016415b1fc70.
    18:35:18,875SERVER POOLED THREAD 19Unexpected error in notifyUser: Connection closed while facing request f9fd70000016415b1fc70:
    18:35:18,875SERVER POOLED THREAD 19Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request f9fd70000016415b1fc70.
    18:35:19,015SERVER POOLED THREAD 5Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request f9fe60000016415b1fc70.
    18:35:19,015SERVER POOLED THREAD 19Unexpected error in notifyUser: Connection closed while facing request f9fe70000016415b1fc70:
    18:35:19,015SERVER POOLED THREAD 19Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request f9fe70000016415b1fc70.
    18:35:19,046SERVER POOLED THREAD 5Unexpected error in notifyUser: Connection closed while facing request f9fe80000016415b1fc70:
    18:35:19,046SERVER POOLED THREAD 5Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request f9fe80000016415b1fc70.
    18:35:19,296SERVER POOLED THREAD 5Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request f9ffb0000016415b1fc70.
    18:35:19,312SERVER POOLED THREAD 29Unexpected error in notifyUser: Connection closed while facing request f9ffc0000016415b1fc70:
    18:35:19,312SERVER POOLED THREAD 29Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request f9ffc0000016415b1fc70.
    18:35:19,343SERVER POOLED THREAD 19Unexpected error in notifyUser: Connection closed while facing request f9ffd0000016415b1fc70:
    18:35:19,673SERVER POOLED THREAD 5Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request fa0140000016415b1fc70.
    18:35:19,673SERVER POOLED THREAD 25Unexpected error in notifyUser: Connection closed while facing request fa0150000016415b1fc70:
    18:35:19,673SERVER POOLED THREAD 25Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request fa0150000016415b1fc70.
    18:35:19,673SERVER POOLED THREAD 5Unexpected error in notifyUser: Connection closed while facing request fa0160000016415b1fc70:
    18:35:19,673SERVER POOLED THREAD 5Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request fa0160000016415b1fc70.
    18:35:19,736SERVER POOLED THREAD 25Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request fa01f0000016415b1fc70.
    18:35:19,736SERVER POOLED THREAD 19Unexpected error in notifyUser: Connection closed while facing request fa0200000016415b1fc70:
    18:35:19,736SERVER POOLED THREAD 19Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request fa0200000016415b1fc70.
    18:35:19,751SERVER POOLED THREAD 25Unexpected error in notifyUser: Connection closed while facing request fa0210000016415b1fc70:
    18:35:19,751SERVER POOLED THREAD 25Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request fa0210000016415b1fc70.
    18:35:19,783SERVER POOLED THREAD 5Unexpected error in notifyUser: Connection closed while facing request fa0220000016415b1fc70:
    18:35:19,783SERVER POOLED THREAD 29Unexpected error in notifyUser: Connection closed while facing request fa0230000016415b1fc70:
    18:35:19,783SERVER POOLED THREAD 5Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request fa0220000016415b1fc70.
    18:35:19,783SERVER POOLED THREAD 29Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request fa0230000016415b1fc70.
    18:35:19,798SERVER POOLED THREAD 19Unexpected error in notifyUser: Connection closed while facing request fa0240000016415b1fc70:
    18:35:19,798SERVER POOLED THREAD 19Failure in request processing: External error while managing the request: Unexpected error in notifyUser: Connection closed while facing request fa0240000016415b1fc70.

  2. #2
    Administrator
    Join Date
    Feb 2012
    Location
    Milano
    Posts
    716
    I can confirm that errors you reported us are due to a connection interruption between the Lightstreamer server and your remote Metadata Adapter.
    Maybe the implementation of notifyUser method in the custom code of your Metatadata Adapter got some errors: it seems that instead of replying to the server your adapter closes the connection in the case of user validation request.

    Given the periodic recurrence, the issue could be caused by a burst of requests that always occurs regularly at the same time.
    What do you do within the notifyUser method? Maybe some request to DB or other external services that can saturate and get stuck in case of overload?

    Regards,
    Giuseppe

  3. #3
    Power Member
    Join Date
    Sep 2013
    Location
    Coimbatore
    Posts
    121
    Hello,
    In this notifyuser method i'm only doing following check

    String origin = (String) httpHeaders.get("origin");
    String un = (String) httpHeaders.get("username");


    if ((origin == null) || VALUES.contains(origin)) {
    if ((un == null) || !un.equalsIgnoreCase("sitekey")) {
    throw new AccessException("Invalid User Access");
    }
    } else {
    if ((un == null) || !un.equalsIgnoreCase("appkey")) {
    throw new AccessException("Invalid User Access");
    }
    }
    didn't check any database check and all. Only checking request(client) coming from my domains. Here we are serving the response to 30 domains. Is it will any issue?
    And also I have one more doubt for this every 30 domains having many clients request and this adapter writing the log about items details in console. Is this will get affect or get any memory leak?
    Last edited by rvkvino; June 21st, 2018 at 04:26 PM.

  4. #4
    Administrator
    Join Date
    Feb 2012
    Location
    Milano
    Posts
    716
    So this means that the Metadata Adapter was already down but didn't reconnect or reconnect with some delays.

    You should look in the log file for the exact moment where the Metadata Adapters go down, and messages like this "Reply receiver '...' stopped". Once identified the moment of the disconnection we will try to understand the causes.

    Regards,
    Giuseppe

  5. #5
    Power Member
    Join Date
    Sep 2013
    Location
    Coimbatore
    Posts
    121
    Hello,
    I have checked my log fully I couldn't find the text as like Reply receiver or stopped.
    I can see Monitor Text log as like below,
    MonitorText|Timer-0 |Total threads = 460, Total heap = 244318208 (free = 22652368), Sessions = 3159 (max = 3175), New sessions = [+20, -11], Connections = 3481 (max = 3575), New connections = [+57, -53], In-pool threads = 128, Active threads = 3, Available threads = 124, Queued tasks = 2, Pool queue wait = 0, NIO write queue = 84, NIO write queue wait = 1872051, NIO write selectors = 32, NIO total selectors = 128, Subscribed items = 47 (for 25462 subscriptions), Inbound throughput = 118.29 updates/s (pre-filtered = 118.29), Outbound throughput = 12841.95 updates/s (7242.07 kbit/s, max = 11693.98), Lost updates = 0 (total = 0), Total bytes sent = 9256882462, Client messages throughput = 0 msgs/s (0 kbit/s, max = 0), Total messages handled = 0, Extra sleep = 0, Notify delay = 0


    MonitorText|Timer-0 |Total threads = 460, Total heap = 240648192 (free = 47881336), Sessions = 3159 (max = 3175), New sessions = [+15, -16], Connections = 3433 (max = 3575), New connections = [+48, -37], In-pool threads = 128, Active threads = 0, Available threads = 128, Queued tasks = 0, Pool queue wait = 1, NIO write queue = 81, NIO write queue wait = 1944088, NIO write selectors = 32, NIO total selectors = 128, Subscribed items = 47 (for 25492 subscriptions), Inbound throughput = 48.21 updates/s (pre-filtered = 48.21), Outbound throughput = 8369.78 updates/s (4749.05 kbit/s, max = 11693.98), Lost updates = 0 (total = 0), Total bytes sent = 9257904075, Client messages throughput = 0 msgs/s (0 kbit/s, max = 0), Total messages handled = 0, Extra sleep = 0, Notify delay = 0


    MonitorText|Timer-0 |Total threads = 460, Total heap = 236453888 (free = 83240984), Sessions = 3121 (max = 3175), New sessions = [+1, -9], Connections = 3496 (max = 3575), New connections = [+87, -28], In-pool threads = 128, Active threads = 34, Available threads = 94, Queued tasks = 395, Pool queue wait = 206, NIO write queue = 59, NIO write queue wait = 2665444, NIO write selectors = 32, NIO total selectors = 128, Subscribed items = 47 (for 25074 subscriptions), Inbound throughput = 71.04 updates/s (pre-filtered = 70.54), Outbound throughput = 10539.99 updates/s (3668.51 kbit/s, max = 11693.98), Lost updates = 0 (total = 0), Total bytes sent = 9260424295, Client messages throughput = 0 msgs/s (0 kbit/s, max = 0), Total messages handled = 0, Extra sleep = 2, Notify delay = 0


    MonitorText|Timer-0 |Total threads = 460, Total heap = 231735296 (free = 16710304), Sessions = 3148 (max = 3175), New sessions = [+16, -12], Connections = 3473 (max = 3575), New connections = [+52, -37], In-pool threads = 128, Active threads = 4, Available threads = 124, Queued tasks = 0, Pool queue wait = 0, NIO write queue = 64, NIO write queue wait = 2457239, NIO write selectors = 32, NIO total selectors = 128, Subscribed items = 47 (for 25363 subscriptions), Inbound throughput = 116.3 updates/s (pre-filtered = 116.3), Outbound throughput = 13418.99 updates/s (6259.6 kbit/s, max = 11693.98), Lost updates = 0 (total = 0), Total bytes sent = 9265303321, Client messages throughput = 0 msgs/s (0 kbit/s, max = 0), Total messages handled = 0, Extra sleep = 0, Notify delay = 0

  6. #6
    Administrator
    Join Date
    Feb 2012
    Location
    Milano
    Posts
    716
    I'm really puzzled that you didn't find any evidence of adapter disconnections.
    Please consider that the disconnection may have happened a long time (several hours) before than the error messages you reported. Furthermore maybe your porxy adapter log settings are not as expected.

    So I think it's better if you focus on your remote Adapter program and check if continues to run non-stop and if there is any loss of communication with the server.

    Regards,
    Giuseppe

  7. #7
    Power Member
    Join Date
    Sep 2013
    Location
    Coimbatore
    Posts
    121
    Hello,
    This issue we have temporarily solved as restart adapter for twice per day. So today we restart the LS server on morning 6.30 A.M and after noon 2 P.M we closed and reopened the remote adapter so we didn't face this issue like connection close on 6 P.M to 7.P.M range. We think any memory leak on this whether LS server or remote adapter. Is there any way to clear the screen of adapter or memory.

  8. #8
    Administrator
    Join Date
    Feb 2012
    Location
    Milano
    Posts
    716
    Hi Vinoth Kumar,

    The only way to influence the memory held by the server is through the 'clear snapshot' request that your Data Adapter can issue for the subscribed Items.
    But it would be preferable to investigate the issue to determine where the leak resides and resolve it.

    Regards,
    Giuseppe

  9. #9
    Power Member
    Join Date
    Sep 2013
    Location
    Coimbatore
    Posts
    121
    Hello,
    i have created an adopter and connecting XXX number of different sites and streaming the data. on each time gap for example from the start time of the adopter on every 12 hours approximately it get stuck in peak hours. We need to know is there any memory limitations for each adopter which we build or any other limitation to be cleared on periodical basis.

    If I restart my adapter 2 times per day then working fine without any issue.

    And also I need to know how dynamically clear snapshot and recreate snapshot. It should be clear if data adapter held my memory issue or if it is any periodically need to clear the snapshot.
    Last edited by rvkvino; June 28th, 2018 at 07:14 AM.

  10. #10
    Administrator
    Join Date
    Feb 2012
    Location
    Milano
    Posts
    716
    Are there in your adapters code some data structure as a queue or a map? Indeed these are potential candidates to generate memory leaks since they can grow indefinitely if not properly managed.
    The Lightstreamer server releases the memory occupied by the snapshot for an Item in two cases:
    1. no more client subscribe the involved Item;
    2. the Data Adapter esplicitily call the clearSnapshot.

    That said, have you tried to check the statistics of the memory occupied looking for the MonitorText logger lines?
    If configured at the trace level it records the information every 2 seconds and can give a general overview of the progress of the memory.

    Regards,
    Giuseppe

 

 

Similar Threads

  1. LS Connection getting disconnect often
    By rvkvino in forum General
    Replies: 3
    Last Post: May 15th, 2018, 08:54 AM
  2. issue in android and ios connection server
    By codesroots in forum Client SDKs
    Replies: 11
    Last Post: March 30th, 2017, 02:13 AM
  3. Replies: 1
    Last Post: February 23rd, 2009, 12:41 PM
  4. Getting through IIS "Basic" authorization process
    By wmolde in forum Client SDKs
    Replies: 14
    Last Post: February 12th, 2008, 10:10 AM
  5. Data Adapter's subscribe() method not getting called
    By CitiMan in forum Adapter SDKs
    Replies: 8
    Last Post: November 30th, 2006, 05:26 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
All times are GMT +1. The time now is 11:07 AM.