Companion Plugin on Server - SCGetInfo huge processor drain?

[DataCruncher]

Hello all - 

My Supercontainer Server works like a charm, and I also use the Companion Plugin in some of the unstored database calculations on the Server. 

For instance, I need to check which of several document locations was uploaded, and the FM unstored calculation field is something along the lines of:

If ( Cases::Client="Test" or Cases::Client="Test 2"; 

   If ( not IsEmpty ( (SCGetInfo("https://domain.com/SuperContainer/Files/REVISION/" & Cases::CaseID & Cases::CUUID))) ; "https://domain.com/SuperContainer/Files/REVISION/" & Cases::CaseID & Cases::CUUID ;    

   If ( not IsEmpty ( (SCGetInfo("https://domain.com/SuperContainer/Files/WordDocs/" & Cases::CaseID & Cases::CUUID))) ; "https://domain.com/SuperContainer/Files/WordDocs/" & Cases::CaseID & Cases::CUUID ;    

 If ( not IsEmpty ( (SCGetInfo("https://domain.com/SuperContainer/Files/AD/" & Cases::CaseID & Cases::CUUID))) ; "https://domain.com/SuperContainer/Files/AD/" & Cases::CaseID & Cases::CUUID ;

      If ( not IsEmpty ( (SCGetInfo("https://domain.com/SuperContainer/Files/TMP/" & Cases::CaseID & Cases::CUUID))) ; "https://domain.com/SuperContainer/Files/TMP/" & Cases::CaseID & Cases::CUUID; 


             If ( not IsEmpty ( (SCGetInfo("https://domain.com/SuperContainer/Files/CG/" & Cases::CaseID & Cases::CUUID))) ; "https://domain.com/SuperContainer/Files/CG/" & Cases::CaseID & Cases::CUUID 
        

)))))


 ;    If ( not IsEmpty ((SCGetInfo("https://domain.com/SuperContainer/Files/REVISION/" & Cases::CaseID & Cases::CUUID))) ; "https://domain.com/SuperContainer/Files/REVISION/" & Cases::CaseID & Cases::CUUID ;  

   If ( not IsEmpty ( (SCGetInfo("https://domain.com/SuperContainer/Files/WordDocs/" & Cases::CaseID & Cases::CUUID))) ; "https://domain.com/SuperContainer/Files/WordDocs/" & Cases::CaseID & Cases::CUUID  ; 

      If ( not IsEmpty ((SCGetInfo("https://domain.com/SuperContainer/Files/TMP/" & Cases::CaseID & Cases::CUUID))) ; "https://domain.com/SuperContainer/Files/TMP/" & Cases::CaseID & Cases::CUUID;

 If ( not IsEmpty ( (SCGetInfo("https://domain.com/SuperContainer/Files/AD/" & Cases::CaseID & Cases::CUUID))) ; "https://domain.com/SuperContainer/Files/AD/" & Cases::CaseID & Cases::CUUID ;


             If ( not IsEmpty ( (SCGetInfo("https://domain.com/SuperContainer/Files/CG/" & Cases::CaseID & Cases::CUUID))) ; "https://domain.com/SuperContainer/Files/CG/" & Cases::CaseID & Cases::CUUID 
        

    

)))) )


)

Now, this particular table already has 250k+ records.

 

I notice the system becoming particularly sluggish to respond whenever that calculation is called upon to the point of the FMS becoming unresponsive for a minute or two. 

Is there any best practice advice on how to cut down FMS server processor load?  I can't tell what the bottleneck is: 

- Something related to SCGetInfo which I presume is some sort of custom function

- Something related to the fact I run many of these SCGetInfo directives for every record

- Or something related to the fact that SuperContainer File Storage is actually on another server in another server center with ping times of 80ms in between. 

 

Perhaps the smart solution would be to only recalculate this if needed? Or only on a subset of records that is otherwise flagged as active? 

I was assuming FMS would be smart enough not to run calculations on records that are not actively used, but the calculation load suggests all records are being calculated? 

 

Any guidance is appreciated - thank you!!

 

 

 

[ryan360Works]

Every SCGetInfo call is a GET request to the SuperContainer server as well as that being in an unstored calc is probably causing the calc to execute several times unnecessarily plus if 250K records are being checked, that's a lot of overhead. How are files being uploaded? via the plugin or a web viewer/browser? If through the plugin, I would just call SCGetInfo as part of the upload process and record that information in a field somewhere. That way you can just check for a value in a field rather than trying to hit the SuperContainer server every time. If you are not uploading via the companion plugin, it might get a bit tricky to do so. 

[DataCruncher]

That's a good idea, it's just that about 85% are uploaded manually through a browser window with no plugin interaction. 

Mind you, all I need is the information file there / not there. I know SCGetInfo gives me whole lot more information that I don't need here, such as Filename, size, upload timestamp. 

 

I really only need a boolean flag as to presence of a file in that folder. Perhaps there is an easier way to test for that other than SCGetInfo?

[DataCruncher]

Ryan: 

 

On 4/27/2020 at 12:51 PM, ryan360Works said:

Every SCGetInfo call is a GET request to the SuperContainer server as well as that being in an unstored calc is probably causing the calc to execute several times unnecessarily plus if 250K records are being checked, that's a lot of overhead. How are files being uploaded? via the plugin or a web viewer/browser? If through the plugin, I would just call SCGetInfo as part of the upload process and record that information in a field somewhere. That way you can just check for a value in a field rather than trying to hit the SuperContainer server every time. If you are not uploading via the companion plugin, it might get a bit tricky to do so. 

 If through the plugin, I would just call SCGetInfo as part of the upload process and record that information in a field somewhere - when you say call SCGetInfo through the plugin - can you please explain where I would need to run SCGetInfo if all the user does is drag and drop a file into the java HTML applet? Can I trigger SCGetInfo through the SuperContainer applet? 

[DataCruncher]

UPDATE: I may be missing something here. I have two pretty much identical FMS 18 instances running on virtualized OS X instances. Like hardware, same software versions. Running the same Filemaker file. One processes SCGetInfo without noticeable delay; the other lags and eventually bugs down the FMS response time to about 35 seconds to render one record. 

 

Is there any log file either on the FMS machine or the SuperContainer server that would point to why SCGetInfo is running slowly on one machine, but not the other? 

 

Am I correct to assume that SCGetInfo is run server-side, rather than on my FMP client? It must be, since the SCGetInfo is part of a calculation field. 

Both FMS machines and the SC server are on the same local subnet even and ping the SuperContainer server in well under 1 msec.

Perhaps my SC Companion plugin is corrupted on one of the machines, or a deprecated java version installed?  

[ryan360Works]

Unless there is an error being recorded I am not sure if the logs files will be helpful. That being said, plugin log file locations can be found here. 

As a test, you can remove the plugin from the equation by sending the GET request through a browser like 

 

<URL to file>?md5=true 

 

Check and make sure you get response headers after sending the request. In Chrome, go to View->Developer ->Developer Tools, Select Network, send the request, select it in the list and then click headers and make sure you got the response. If the issue is with the plugin I would suspect that the responses should take the same time on both machines. 

 

Alternatively, if you are familiar with how to use cURL in terminal/CMD you can do the same test.