#4670: Request modification, VDT Squid startup script

#

Tue Jan 13 10:04:27 2009

timm@fnal.gov - Ticket created

Subject:	Request modification, VDT Squid startup script
Date:	Tue, 13 Jan 2009 10:03:22 -0600 (CST)
To:	vdt-support@OPENSCIENCEGRID.ORG
From:	Steven Timm <timm@fnal.gov>

Download (untitled) / with headers
text/plain 687b

FermiGrid has found that when squid is restarted and/or
the machine is rebooted, it is usually necessary and always good
practice to zero the squid cache, also to start it with more
file descriptors than the default. I have attached to this E-mail
the modifications that we made. Can these please be
added to the VDT (adjusting for hard-wired directories as necessary).

Thanks

Steven Timm

--
------------------------------------------------------------------
Steven C. Timm, Ph.D (630) 840-8525
timm@fnal.gov http://home.fnal.gov/~timm/
Fermilab Computing Division, Scientific Computing Facilities,
Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader.

Download squid_init.d
text/plain 1.6k

#	Tue Jan 13 10:12:04 2009	roy - Priority changed from (no value) to '3'

#	Tue Jan 13 10:12:56 2009	roy - Correspondence added		[Reply]
	Download (untitled) / with headers text/plain 642b On Tue Jan 13 10:04:27 2009, timm@fnal.gov wrote: > > FermiGrid has found that when squid is restarted and/or > the machine is rebooted, it is usually necessary and always good > practice to zero the squid cache, also to start it with more > file descriptors than the default. Can you explain a bit more why it is necessary and/or good? You have more operational experience with Squid, so I would like to understand a bit better. -alain ----------------------------------------------------------------- Alain Roy vdt-support@opensciencegrid.org VDT Support http://vdt.cs.wisc.edu/support.html

#	Tue Jan 13 10:12:58 2009	RT_System - Status changed from 'new' to 'open'

#	Tue Jan 13 10:13:17 2009	roy - Given to roy

#

Tue Jan 13 10:38:25 2009

timm@fnal.gov - Correspondence added

[Reply]

Subject:	Re: [vdt-support #4670] Request modification, VDT Squid startup script
Date:	Tue, 13 Jan 2009 10:34:40 -0600 (CST)
To:	Alain Roy via RT <vdt-support@OPENSCIENCEGRID.ORG>
From:	Steven Timm <timm@fnal.gov>

Download (untitled) / with headers
text/plain 1.3k

On Tue, 13 Jan 2009, Alain Roy via RT wrote:

> On Tue Jan 13 10:04:27 2009, timm@fnal.gov wrote:

>>
>> FermiGrid has found that when squid is restarted and/or
>> the machine is rebooted, it is usually necessary and always good
>> practice to zero the squid cache, also to start it with more
>> file descriptors than the default.

>
> Can you explain a bit more why it is necessary and/or good? You have
> more operational experience with Squid, so I would like to understand a
> bit better.

Our experience shows that unless this is done, squid does not start up
clean, you either get error messages and it refuses to start at all,
or it starts up quietly and it is in a degraded mode.

We compared notes with the people who maintain the Frontier database
servers for CMS and they are doing the same thing, namely wiping
the cache directory of files and zeroing the squid cache on every restart.

Steve Timm

>
> -alain
>
> -----------------------------------------------------------------
> Alain Roy vdt-support@opensciencegrid.org
> VDT Support http://vdt.cs.wisc.edu/support.html
>
>
>

--
------------------------------------------------------------------
Steven C. Timm, Ph.D (630) 840-8525
timm@fnal.gov http://home.fnal.gov/~timm/
Fermilab Computing Division, Scientific Computing Facilities,
Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader.

#	Tue Jan 13 10:42:22 2009	roy - Given to adesmet

#

Tue Jan 13 10:48:38 2009

roy - Correspondence added

[Reply]

Subject:	Re: [vdt-support #4670] Request modification, VDT Squid startup script
Date:	Tue, 13 Jan 2009 10:40:52 -0600
To:	vdt-support@OPENSCIENCEGRID.ORG
From:	Alain Roy <roy@cs.wisc.edu>

Download (untitled) / with headers
text/plain 577b

On Jan 13, 2009, at 10:38 AM, timm@fnal.gov via RT wrote:

> Our experience shows that unless this is done, squid does not start up
> clean, you either get error messages and it refuses to start at all,
> or it starts up quietly and it is in a degraded mode.
>
> We compared notes with the people who maintain the Frontier database
> servers for CMS and they are doing the same thing, namely wiping
> the cache directory of files and zeroing the squid cache on every
> restart.

It sounds like a good idea. We'll try to schedule this work in the
next month.

Thanks,
-alain

#	Wed Jan 14 17:21:43 2009	adesmet - Correspondence added		[Reply]
	Download (untitled) / with headers text/plain 518b I'm working on integrating your changes. The cache flush should be straightforward. I do have one question. You wrote: > FermiGrid has found that when squid is restarted ... also to start it with more file descriptors than the default. The VDT has set the file descriptor limit to 16384 for a bit shy of a year now. This matches the init script you sent along. So it would appear no work is necessary. Are you not seeing this on your end? If you're not seeing it, then perhaps we have a bug we need to fix.

#	Wed Jan 14 17:30:40 2009	adesmet - Comments added		[Reply]
	Download (untitled) / with headers text/plain 921b What will the impact be on other users? This may cause serious slowdowns as Squid will no longer be able to return cached results on a restart until it builds up that cache. Specific options that Timm added, but didn't mention. Find out why they were added, and if they want them added. -D - Disable DNS tests (allow Squid to start without a working DNS server) - could be done via dns_testnames in squid.conf - Seems like a silly test for our users in the first place. What's the harm of running without a DNS server? Probably more useful for ISPs, where if Squid "breaks" but DNS works for the end user, the user will be angry and will want Squid turned off. On the other hand, sans Squid, they have nothing. -F - Refuse requests until storage metadata is rebuilt - Seems harsh. (Although admittedly brief given the context) - Intended to speed up rebuild. What is there to speed up? It's empty! -S - ??

#

Thu Jan 15 09:17:29 2009

timm@fnal.gov - Correspondence added

[Reply]

Subject:	Re: [vdt-support #4670] Request modification, VDT Squid startup script
Date:	Thu, 15 Jan 2009 09:16:34 -0600 (CST)
To:	Alan De Smet via RT <vdt-support@OPENSCIENCEGRID.ORG>
From:	Steven Timm <timm@fnal.gov>

Download (untitled) / with headers
text/plain 1k

On Wed, 14 Jan 2009, Alan De Smet via RT wrote:

> I'm working on integrating your changes. The cache flush should be
> straightforward. I do have one question. You wrote:
>

>> FermiGrid has found that when squid is restarted ... also to start it

> with more file descriptors than the default.
>
> The VDT has set the file descriptor limit to 16384 for a bit shy of a
> year now. This matches the init script you sent along. So it would
> appear no work is necessary. Are you not seeing this on your end? If
> you're not seeing it, then perhaps we have a bug we need to fix.

No--I just thought that it was a change that we made, probably remembering
that it was us who requested that change in the first place.
As long as it's in the default already, that is fine.

Thanks

Steve Timm

>
>
>

--
------------------------------------------------------------------
Steven C. Timm, Ph.D (630) 840-8525
timm@fnal.gov http://home.fnal.gov/~timm/
Fermilab Computing Division, Scientific Computing Facilities,
Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader.

#	Thu Jan 15 17:32:56 2009	adesmet - Correspondence added		[Reply]
	Download (untitled) / with headers text/plain 38b -S - double check swap during rebuild.

#	Thu Jan 15 17:37:14 2009	adesmet - Correspondence added		[Reply]
	Download (untitled) / with headers text/plain 549b Sorry, one more question. The updated init script includes the line: ${SQUID} -z ${SQUID_OPTIONS## -DFS} SQUID_OPTIONS looks something like "-f /path/to/squid.conf -DFS", and the "## -DFS" is intended to remove the -DFS. However, "##" removes strings found at the start, so it has no effect. As such "%%" (which removes strings found at the end) seems more correct. So I plan on actually committing a line like: ${SQUID} -z ${SQUID_OPTIONS%% -DFS} Is this correct? If not, could you explain what is going on that I don't udnerstand?

#

Thu Jan 15 19:31:17 2009

timm@fnal.gov - Correspondence added

[Reply]

Subject:	Re: [vdt-support #4670] Request modification, VDT Squid startup script
Date:	Thu, 15 Jan 2009 19:30:20 -0600 (CST)
To:	Alan De Smet via RT <vdt-support@OPENSCIENCEGRID.ORG>
From:	Steven Timm <timm@fnal.gov>

Download (untitled) / with headers
text/plain 1k

I have forwarded your question to Keith Chadwick, who made the initial
mod, to see if he remembers why he did it that way.

Steve Timm

On Thu, 15 Jan 2009, Alan De Smet via RT wrote:

> Sorry, one more question.
>
> The updated init script includes the line:
>
> ${SQUID} -z ${SQUID_OPTIONS## -DFS}
>
> SQUID_OPTIONS looks something like "-f /path/to/squid.conf -DFS", and
> the "## -DFS" is intended to remove the -DFS. However, "##" removes
> strings found at the start, so it has no effect. As such "%%" (which
> removes strings found at the end) seems more correct. So I plan on
> actually committing a line like:
>
> ${SQUID} -z ${SQUID_OPTIONS%% -DFS}
>
> Is this correct? If not, could you explain what is going on that I
> don't udnerstand?
>
>

--
------------------------------------------------------------------
Steven C. Timm, Ph.D (630) 840-8525
timm@fnal.gov http://home.fnal.gov/~timm/
Fermilab Computing Division, Scientific Computing Facilities,
Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader.

#

Wed Jan 21 17:09:23 2009

adesmet - Comments added

[Reply]

Subject:	[vdt-support #4670] SVN commit, rev 8612
To:	vdt-support@cs.wisc.edu
From:	adesmet@cs.wisc.edu

Download (untitled) / with headers
text/plain 478b

Commit comment:

Scrub Squid's cache on startup. In the process move path to squid.conf into a shared location so ensure it stays the same.

This is reported to fix problems with Squid installations that shutdown uncleanly. Reportedly Squid doesn't recover well, and either runs in a degraded mode, or fails to run at all.

Changed files:
U vdt/branches/vdt-1.10.1/Configure-Squid/vdt/setup/configure_squid

To generate a diff:
svn diff -c 8612 file:///p/vdt/workspace/svn

#	Mon Jan 26 13:38:01 2009	adesmet - Correspondence added		[Reply]
	Download (untitled) / with headers text/plain 336b VDT 1.10.1s includes the cache-resetting code in the Squid init script. It doesn't include the "## -DFS". As best I can tell, it doesn't do anything, and Squid appears to work as expected. If Keith Chadwick comes back and reports that it's necessary, we'll see about adding it. Otherwise I think everything requested is now present.

#	Mon Jan 26 13:38:21 2009	adesmet - Status changed from 'open' to 'resolved'