#3687: per_source limit, xinetd globus-gatekeeper

#

Wed Jul 23 08:56:05 2008

timm@fnal.gov - Ticket created

Subject:	per_source limit, xinetd globus-gatekeeper
Date:	Wed, 23 Jul 2008 08:52:04 -0500 (CDT)
To:	vdt-support@OPENSCIENCEGRID.ORG
From:	Steven Timm <timm@fnal.gov>

Download (untitled) / with headers
text/plain 1.5k

Under RHEL5/SL5 there is a new default limit in
/etc/xinetd.conf. It is called per_source and by default
allows only 10 simultaneous connections to any given
xinetd-managed port from any one IP number.

If you are dealing with a busy condor-G client talking
to a globus gatekeeper that has just come on the air,
this can lead to globus-gatekeeper connections being denied by xinetd
with the message:

Jul 6 04:05:23 fcdfosg3 xinetd[3070]: FAIL: globus-gatekeeper
per_source_limit from=131.225.211.230

The result can be that the Condor-G client will consider
the remote gatekeeper to be DOWN if it fails enough, and refuse
to submit any more jobs to the resource.

This can be corrected by adding a setting to the xinetd
file for globus-gatekeeper

per_source 300

I picked 300 as an ad-hoc number that was bigger than 10 and
it was big enough to clear a backlog of 1500 simultaneous
jobs on my submit gateway.
At the limit the vdt documentation on the globus-gatekeeper should
mention that this setting can be made.

This problem was actually with us in all VDT's including
the two SL5/vdt 1.8.1 installations I had for months, but
I only noticed it at the vdt 1.10.1 upgrade when my three
busiest gatekeepers at FermiGrid were all upgraded to SL5.

Steve Timm

--
------------------------------------------------------------------
Steven C. Timm, Ph.D (630) 840-8525
timm@fnal.gov http://home.fnal.gov/~timm/
Fermilab Computing Division, Scientific Computing Facilities,
Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader.

#	Tue Jul 29 22:29:31 2008	roy - Correspondence added		[Reply]
	Download (untitled) / with headers text/plain 972b > Under RHEL5/SL5 there is a new default limit in > /etc/xinetd.conf. It is called per_source and by default > allows only 10 simultaneous connections to any given > xinetd-managed port from any one IP number. Thanks for the tip, Steve! I have two questions: 1) Do you know how I can reliably detect if xinetd supports this feature? One option is that we can set this for people when appropriate? 2) Do you know if I always set this for xinetd, will it cause a problem for older versions of xinetd? Another option is that we can set this for all installations if it doesn't cause problems when it's an unrecognized option. I'm happy to document it, but I'm worried that the documentation will get buried, so I would rather set the option if I could. Thanks, -alain ----------------------------------------------------------------- Alain Roy vdt-support@opensciencegrid.org VDT Support http://vdt.cs.wisc.edu/support.html

#	Tue Jul 29 22:29:32 2008	RT_System - Status changed from 'new' to 'open'

#	Tue Jul 29 22:29:32 2008	roy - Given to roy

#	Tue Jul 29 22:30:37 2008	roy - FinalPriority changed from (no value) to '3'

#	Tue Jul 29 22:30:37 2008	roy - Priority changed from (no value) to '3'

#	Tue Jul 29 22:30:38 2008	roy - Fix scheduled CUR added

#

Tue Jul 29 22:45:03 2008

timm@fnal.gov - Correspondence added

[Reply]

Subject:	Re: [vdt-support #3687] per_source limit, xinetd globus-gatekeeper
Date:	Tue, 29 Jul 2008 22:39:44 -0500 (CDT)
To:	Alain Roy via RT <vdt-support@OPENSCIENCEGRID.ORG>
From:	Steven Timm <timm@fnal.gov>

Download (untitled) / with headers
text/plain 1.6k

------------------------------------------------------------------
Steven C. Timm, Ph.D (630) 840-8525
timm@fnal.gov http://home.fnal.gov/~timm/
Fermilab Computing Division, Scientific Computing Facilities,
Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader.

On Tue, 29 Jul 2008, Alain Roy via RT wrote:

>> Under RHEL5/SL5 there is a new default limit in
>> /etc/xinetd.conf. It is called per_source and by default
>> allows only 10 simultaneous connections to any given
>> xinetd-managed port from any one IP number.

>
> Thanks for the tip, Steve!
>
> I have two questions:
>
> 1) Do you know how I can reliably detect if xinetd supports this
> feature? One option is that we can set this for people when appropriate?

I don't know for sure, but on the xinetd.conf file the per_source
default is listed when it is enabled.

>
> 2) Do you know if I always set this for xinetd, will it cause a problem
> for older versions of xinetd? Another option is that we can set this for
> all installations if it doesn't cause problems when it's an unrecognized
> option.
>

don't know but it is easy to add it to SL3 or SL4 and see.

Steve Timm

> I'm happy to document it, but I'm worried that the documentation will
> get buried, so I would rather set the option if I could.
>
> Thanks,
> -alain
>
> -----------------------------------------------------------------
> Alain Roy vdt-support@opensciencegrid.org
> VDT Support http://vdt.cs.wisc.edu/support.html
>
>
> --
> View ticket at <http://vdt.cs.wisc.edu/rt/Ticket/Display.html?user=guest&pass=guest&id=3687>
> VDT Support, vdt-support@ivdgl.org
>

#

Tue Jul 29 23:05:20 2008

roy - Correspondence added

[Reply]

Subject:	Re: [vdt-support #3687] per_source limit, xinetd globus-gatekeeper
Date:	Tue, 29 Jul 2008 23:03:24 -0500
To:	vdt-support@OPENSCIENCEGRID.ORG
From:	Alain Roy <roy@cs.wisc.edu>

Download (untitled) / with headers
text/plain 410b

>> 2) Do you know if I always set this for xinetd, will it cause a
>> problem
>> for older versions of xinetd? Another option is that we can set
>> this for
>> all installations if it doesn't cause problems when it's an
>> unrecognized
>> option.
>>

> don't know but it is easy to add it to SL3 or SL4 and see.

Yup, we'll give it a shot when we have time.

Thanks! We'll try to act on this soon.
-alain

#	Tue Sep 02 15:55:49 2008	roy - Comments added		[Reply]
	Download (untitled) / with headers text/plain 126b Alan--can you investigate this? I'm not sure what the right thing to do is. Can you investigate and propose? Thanks, -alain

#	Tue Sep 02 15:55:56 2008	roy - Given to adesmet

#	Thu Sep 04 11:37:00 2008	adesmet - Comments added		[Reply]
	Download (untitled) / with headers text/plain 260b Investigating. My goals: - Is this widely portable? (If so, we can just turn it on everywhere.) - If it's not portable, can we detect the functionality? (If so, we can turn it on conditionally.) - Failing the above, we're probably reduced to documenting it.

#	Thu Sep 04 14:06:37 2008	adesmet - Comments added		[Reply]
	Download (untitled) / with headers text/plain 376b The per_source option (and support for UNLIMITED) dates back at least as far as February 19, 2003, which is when the current xinetd CVS repository was started. Support seems likely to be pervasive. I'll still do a quick check of our platforms to confirm that no one is running an antique version. While it seems implausible, Mac OS X is shipping a surprisingly old version.

#	Thu Sep 04 15:28:49 2008	adesmet - Comments added		[Reply]
	Download (untitled) / with headers text/plain 482b Every platform we currently support ships at least 2.3.12, from August 2003. (I have no idea why they're shipping such an old version.) The binaries appear to include the relevant symbols. I think we're good everywhere. Now the question is "how big". xinetd does allow for UNLIMITED, but limits do seem like a good thing. 5 is obviously too small. 300 was apparently good enough for a real use case. So for now I'll go with 300; changing it in the future should easy enough.

#	Wed Sep 24 15:49:56 2008	adesmet - Comments added		[Reply]
	Download (untitled) / with headers text/plain 23b Implemented for 1.10.1k

#	Wed Sep 24 15:50:07 2008	adesmet - Status changed from 'open' to 'resolved'