Skip Menu | Logged in as guest | Logout
 
Ticket metadata
Id: 3940
Status: resolved
Priority: 3/0
Queue: vdt-support

Fixed in: 1.10.1u
Fix scheduled: CUR

Owner: Tim Cartwright
Requestors: cgw@hep.uchicago.edu
Marco Mambelli
saewill@iupui.edu
Cc:
AdminCc:

More about cgw@hep.uchicago.edu
Comments about this user:
No comment entered about this user
This user's 10 highest priority tickets:
Groups this user belongs to:
  • Everyone
  • Unprivileged

More about saewill@iupui.edu
Comments about this user:
No comment entered about this user
This user's 10 highest priority tickets:
Groups this user belongs to:
  • Unprivileged
  • Everyone

New reminder:

Created: Thu Sep 11 14:57:01 2008
Starts: Not set
Started: Not set
Last Contact: Wed Mar 04 11:56:36 2009
Due: Not set
Closed: Wed Mar 04 12:12:24 2009
Updated: Thu Sep 24 07:31:41 2009 by guest



History Brief headersFull headers
CC: mwt2-core-l@LISTSERV.INDIANA.EDU
Subject: Problem with VDT and DQ2
Date: Thu, 11 Sep 2008 14:48:03 -0500 (CDT)
To: vdt-support@OPENSCIENCEGRID.ORG
From: Marco Mambelli <marco@hep.uchicago.edu>
Download (untitled) / with headers
text/plain 1.1k
Dear VDT support,
I have 2 installations of wlcg-client (including vdt-client and little
more), VDT version 1.10.1h
- one installed as root
- one installed as private user
they share a common certificate directory (/etc/grid-security) not
maintained by any of the 2

The second one works fine

The first one gives an error when a DQ2 (an ATLAS utility) is invoked:
> dq2-register-subscription mc08.105009.J0_pythia_jetjet.recon.ESD.e344_s456_r456 UCT3
Error: [(py)CURL][FATAL] DQ2 internal exception [

(77, 'error setting certificate verify locations:\n CAfile: /home/condor/execute/dir_7315/userdir/curl/share/curl/curl-ca-bundle.crt\n
CApath: /share/wlcg-client/globus/TRUSTED_CA\n')]

[http://atlddmcat.cern.ch:80//dq2https://atlddmcat.cern.ch:443//dq2][curl
error code:77]

The problem seems a reference to a curl certificate in a directory not
existent
curl-ca-bundle.crt is in $VDT_LOCATION/curl/share/curl/curl-ca-bundle.crt,
not in /home/condor that is empty
Any idea how to troubleshoot further?
Which is the difference betwen the user/root installations?
normal curl invocation (http/https on command line) work

Thank you,
Marco
CC: vdt-support@OPENSCIENCEGRID.ORG, mwt2-core-l@listserv.indiana.edu
Subject: re: Problem with VDT and DQ2
Date: Thu, 11 Sep 2008 14:51:09 -0500
To: Marco Mambelli <marco@hep.uchicago.edu>
From: Charles Waldman <cgw@hep.uchicago.edu>
Download (untitled) / with headers
text/plain 1.3k
Another relevant detail: the non-working install was done via
"pacman -update". Trying again with a clean install dir.

- Charles

Marco Mambelli writes:
> Dear VDT support,
> I have 2 installations of wlcg-client (including vdt-client and little
> more), VDT version 1.10.1h
> - one installed as root
> - one installed as private user
> they share a common certificate directory (/etc/grid-security) not
> maintained by any of the 2
>
> The second one works fine
>
> The first one gives an error when a DQ2 (an ATLAS utility) is invoked:
> > dq2-register-subscription mc08.105009.J0_pythia_jetjet.recon.ESD.e344_s456_r456 UCT3
> Error: [(py)CURL][FATAL] DQ2 internal exception [
>
> (77, 'error setting certificate verify locations:\n CAfile: /home/condor/execute/dir_7315/userdir/curl/share/curl/curl-ca-bundle.crt\n
> CApath: /share/wlcg-client/globus/TRUSTED_CA\n')]
>
> [http://atlddmcat.cern.ch:80//dq2https://atlddmcat.cern.ch:443//dq2][curl
> error code:77]
>
> The problem seems a reference to a curl certificate in a directory not
> existent
> curl-ca-bundle.crt is in $VDT_LOCATION/curl/share/curl/curl-ca-bundle.crt,
> not in /home/condor that is empty
> Any idea how to troubleshoot further?
> Which is the difference betwen the user/root installations?
> normal curl invocation (http/https on command line) work
>
> Thank you,
> Marco
CC: Alain Roy <roy@cs.wisc.edu>, vdt-support@OPENSCIENCEGRID.ORG, mwt2-core-l@LISTSERV.INDIANA.EDU
Subject: Problem with VDT and DQ2
Date: Thu, 11 Sep 2008 15:59:12 -0500
To: Marco Mambelli <marco@hep.uchicago.edu>
From: Charles Waldman <cgw@hep.uchicago.edu>
Download (untitled) / with headers
text/plain 2.6k
Marco - after doing a clean installation rather than pacman -update,
and creating a symlink /share/wlcg-client/globus/TRUSTED_CA ->
/etc/grid-security/certificates, the dq2-register-subscription
command works for me. I created the symlink because the message
below says:

> CApath: /share/wlcg-client/globus/TRUSTED_CA

and this file does not exist (certs are maintaned separately
in /etc/grid-security/certificates).

However, removing this symlink and retrying the test, it still
works, so I suspect that the fact that
/share/wlcg-client/globus/TRUSTED_CA does not actually exist is
not the real problem. It certainly adds confusion to see this
path mentioned in the diag output, as well as
/home/condor/execute/dir_7315/userdir/curl/share/curl/curl-ca-bundle.crt
which I think is a build artefact and really should not appear in
VDT-shipped binaries.

I suspect that your test failure is a result of env. var settings.
When I try the following commands from a clean login shell, I do
not get the exception you report.

uct3-edge3$ . /share/wlcg-client/setup.sh
uct3-edge3$ voms-proxy-init -voms atlas:/usatlas/Role=production
uct3-edge3$ . /share/dq2-client/setup.sh
uct3-edge3$ dq2-register-subscription mc08.105009.J0_pythia_jetjet.recon.ESD.e344_s456_r456 UCT3

Error: [USER][OTHER] A dataset subscription for 59d2b886-4dcb-4755-b63c-789d5e46961f already exists at 2025!

This is an expected error, not a failure.

- Charles



Marco Mambelli writes:
> Dear VDT support,
> I have 2 installations of wlcg-client (including vdt-client and little
> more), VDT version 1.10.1h
> - one installed as root
> - one installed as private user
> they share a common certificate directory (/etc/grid-security) not
> maintained by any of the 2
>
> The second one works fine
>
> The first one gives an error when a DQ2 (an ATLAS utility) is invoked:
> > dq2-register-subscription mc08.105009.J0_pythia_jetjet.recon.ESD.e344_s456_r456 UCT3
> Error: [(py)CURL][FATAL] DQ2 internal exception [
>
> (77, 'error setting certificate verify locations:\n CAfile: /home/condor/execute/dir_7315/userdir/curl/share/curl/curl-ca-bundle.crt\n
> CApath: /share/wlcg-client/globus/TRUSTED_CA\n')]
>
> [http://atlddmcat.cern.ch:80//dq2https://atlddmcat.cern.ch:443//dq2][curl
> error code:77]
>
> The problem seems a reference to a curl certificate in a directory not
> existent
> curl-ca-bundle.crt is in $VDT_LOCATION/curl/share/curl/curl-ca-bundle.crt,
> not in /home/condor that is empty
> Any idea how to troubleshoot further?
> Which is the difference betwen the user/root installations?
> normal curl invocation (http/https on command line) work
>
> Thank you,
> Marco
Subject: [vdt-support #3942] AutoReply: Problem with VDT and DQ2
Date: Thu, 11 Sep 2008 16:15:45 -0500
To: vdt-support@OPENSCIENCEGRID.ORG
From: Charles Waldman <cgw@hep.uchicago.edu>
Download (untitled) / with headers
text/plain 3.3k
Further note: the symlink /share/wlcg-client/globus/TRUSTED_CA ->
/etc/grid-security/certificates is necessary after all - without
this, "voms-proxy-init" fails with:

unable to access trusted certificates
in:x509_cert_dir=/share/wlcg-client/globus/TRUSTED_CA

Not sure why this link did not get created during the installation,
very possibly an error on my part.

- Charles


> vdt-support@opensciencegrid.org
>
> -------------------------------------------------------------------------
> Marco - after doing a clean installation rather than pacman -update,
> and creating a symlink /share/wlcg-client/globus/TRUSTED_CA ->
> /etc/grid-security/certificates, the dq2-register-subscription
> command works for me. I created the symlink because the message
> below says:
>
> > CApath: /share/wlcg-client/globus/TRUSTED_CA
>
> and this file does not exist (certs are maintaned separately
> in /etc/grid-security/certificates).
>
> However, removing this symlink and retrying the test, it still
> works, so I suspect that the fact that
> /share/wlcg-client/globus/TRUSTED_CA does not actually exist is
> not the real problem. It certainly adds confusion to see this
> path mentioned in the diag output, as well as
> /home/condor/execute/dir_7315/userdir/curl/share/curl/curl-ca-bundle.crt
> which I think is a build artefact and really should not appear in
> VDT-shipped binaries.
>
> I suspect that your test failure is a result of env. var settings.
> When I try the following commands from a clean login shell, I do
> not get the exception you report.
>
> uct3-edge3$ . /share/wlcg-client/setup.sh
> uct3-edge3$ voms-proxy-init -voms atlas:/usatlas/Role=production
> uct3-edge3$ . /share/dq2-client/setup.sh
> uct3-edge3$ dq2-register-subscription mc08.105009.J0_pythia_jetjet.recon.ESD.e344_s456_r456 UCT3
>
> Error: [USER][OTHER] A dataset subscription for 59d2b886-4dcb-4755-b63c-789d5e46961f already exists at 2025!
>
> This is an expected error, not a failure.
>
> - Charles
>
>
>
> Marco Mambelli writes:
> > Dear VDT support,
> > I have 2 installations of wlcg-client (including vdt-client and little
> > more), VDT version 1.10.1h
> > - one installed as root
> > - one installed as private user
> > they share a common certificate directory (/etc/grid-security) not
> > maintained by any of the 2
> >
> > The second one works fine
> >
> > The first one gives an error when a DQ2 (an ATLAS utility) is invoked:
> > > dq2-register-subscription mc08.105009.J0_pythia_jetjet.recon.ESD.e344_s456_r456 UCT3
> > Error: [(py)CURL][FATAL] DQ2 internal exception [
> >
> > (77, 'error setting certificate verify locations:\n CAfile: /home/condor/execute/dir_7315/userdir/curl/share/curl/curl-ca-bundle.crt\n
> > CApath: /share/wlcg-client/globus/TRUSTED_CA\n')]
> >
> > [http://atlddmcat.cern.ch:80//dq2https://atlddmcat.cern.ch:443//dq2][curl
> > error code:77]
> >
> > The problem seems a reference to a curl certificate in a directory not
> > existent
> > curl-ca-bundle.crt is in $VDT_LOCATION/curl/share/curl/curl-ca-bundle.crt,
> > not in /home/condor that is empty
> > Any idea how to troubleshoot further?
> > Which is the difference betwen the user/root installations?
> > normal curl invocation (http/https on command line) work
> >
> > Thank you,
> > Marco
CC: vdt-support@OPENSCIENCEGRID.ORG, mwt2-core-l@listserv.indiana.edu
Subject: re: Problem with VDT and DQ2 [vdt-support #3942]
Date: Thu, 11 Sep 2008 22:10:35 -0500
To: Marco Mambelli <marco@hep.uchicago.edu>
From: Charles Waldman <cgw@hep.uchicago.edu>
Download (untitled) / with headers
text/plain 2.1k
Marco -

The problems you report are all due to site-level problems and nothing
with the VDT installer. I've straightened out the issues on
uct3-edge5 and uct3-edge3 - a file
/usr/lib/python2.3/site-packages/pycurl.so was present on uct3-edge5
but not edge3. This file was put there by me when I was trying
to get Hiro's "deleteDatasetLocal2.py" script to work - this has
some prerequisistes that are not compatible with the "dq2-clients"
package, and these files were keeping the dq2-clients pkg from
working. So I moved pycurl.so to
'/usr/lib/python-2.3/site-packages-disabled' and now the
dq2-register-subscription works.

I figured this out by using a whole bunch of strace and diff commands,
it wasn't too difficult.

Even if VDT installer supplies all needed prerequisites, if there
are non-compatible libs installed in "system" locations, things will
sometimes break.

I think we can close out this VDT ticket.

- Charles



Marco Mambelli writes:
> Dear VDT support,
> I have 2 installations of wlcg-client (including vdt-client and little
> more), VDT version 1.10.1h
> - one installed as root
> - one installed as private user
> they share a common certificate directory (/etc/grid-security) not
> maintained by any of the 2
>
> The second one works fine
>
> The first one gives an error when a DQ2 (an ATLAS utility) is invoked:
> > dq2-register-subscription mc08.105009.J0_pythia_jetjet.recon.ESD.e344_s456_r456 UCT3
> Error: [(py)CURL][FATAL] DQ2 internal exception [
>
> (77, 'error setting certificate verify locations:\n CAfile: /home/condor/execute/dir_7315/userdir/curl/share/curl/curl-ca-bundle.crt\n
> CApath: /share/wlcg-client/globus/TRUSTED_CA\n')]
>
> [http://atlddmcat.cern.ch:80//dq2https://atlddmcat.cern.ch:443//dq2][curl
> error code:77]
>
> The problem seems a reference to a curl certificate in a directory not
> existent
> curl-ca-bundle.crt is in $VDT_LOCATION/curl/share/curl/curl-ca-bundle.crt,
> not in /home/condor that is empty
> Any idea how to troubleshoot further?
> Which is the difference betwen the user/root installations?
> normal curl invocation (http/https on command line) work
>
> Thank you,
> Marco
CC: mwt2-core-l@listserv.indiana.edu, Alain Roy <roy@cs.wisc.edu>
Subject: [vdt-support #3942] dq2-client is incompatible with dq2-site-services
Date: Thu, 11 Sep 2008 22:25:41 -0500
To: Marco Mambelli <marco@hep.uchicago.edu>, vdt-support@OPENSCIENCEGRID.ORG
From: Charles Waldman <cgw@hep.uchicago.edu>
Download (untitled) / with headers
text/plain 1018b
Not sure how serious of a problem this is, but if a host
is running dq2 site services, then the dq2-clients package
will not work - dq2 site services is the source of the
/usr/lib/pythonXX/pycurl.so file... installed via an
RPM package "python-curl-7.16.4-1". Here's a case where
the VDT installer would perhaps benefit from better
integration with the system RPM database, if it could detect
that an incompatible pycurl.so were installed. Even nicer
would be if somehow dq2-site-services and the dq2-clients

But it's tricky, especially when things get installed via NFS.
The server where this got installed did not have the dq2-site-services
package installed, so even if the VDT installer tested for the
incompatible pycurl.so, it would not have been found. It was only
when this got NFS-mounted on a client that had the dq2-site-services
RPMs installed that there was a problem. Not sure what, if anything,
could have been done by the VDT installer to prevent this. It's a
tricky problem.

- Charles
CC: vdt-support@OPENSCIENCEGRID.ORG, mwt2-core-l@listserv.indiana.edu, Alain Roy <roy@cs.wisc.edu>
Subject: Re: [vdt-support #3942] dq2-client is incompatible with dq2-site-services
Date: Fri, 12 Sep 2008 14:54:20 -0500 (CDT)
To: Charles Waldman <cgw@hep.uchicago.edu>
From: Marco Mambelli <marco@hep.uchicago.edu>
Download (untitled) / with headers
text/plain 2.7k
Hi Alain, Charles,
I think I found the problem.

There is an error in VDT as it is distributed and DQ2+pycurl triggered the
error
It is a config error, not an incompatibility error

Error:
curl has a seried of options (settable with curl_easy_setopt or
curl_obj.setopt() in python). 2 of these optins refer to CA :
CURLOPT_CAINFO - file with trusted certs
CURLOPT_CAPATH - directory with trusted certs

In VDT curl distribution CURLOPT_CAINFO is set to:
/home/condor/execute/dir_7315/userdir/curl/share/curl/curl-ca-bundle.crt
instead of $VDT_LOCATION/curl/share/curl/curl-ca-bundle.crt
This default setting comes from curl/lib/libcurl.so.4 in VDT 1.10.1h (and
earlier)
This file contains certs trusted by curl developers. It is actually not
used, so the option could be even unset.
I think ptobably at compile time you have control over it.

How it is triggered:
DQ2 uses curl with a python wrapper
if pycurl is there it is preferred, else curl (command) is used
The addition of pycurl in pythonpath triggered the use of it.
Pycurl is using libcurl from VDT: libcurl.so.4 =>
/ecache/share/wlcg-client080911/curl/lib/libcurl.so.4 (0x00be1000)
When using pycurl DQ2 is setting CURLOPT_CAPATH to the globus cert dir and
it is ignoring CURLOPT_CAINFO.
When curl library is verifying the CAs it fails because the file
/home/condor/execute/dir_7315/userdir/curl/share/curl/curl-ca-bundle.crt
is not found.


Solution:
compile curl in VDT with the correct path (or without CURLOPT_CAINFO)


Workaround:
create a link from the /home/condor/ directory so that the file is found
add a line like
csec.unsetopt(pycurl.CAINFO)
in the DQ2 code to unset the CURLOPT_CAINFO

Cheers,
Marco



On Thu, 11 Sep 2008, Charles Waldman wrote:

>
> Not sure how serious of a problem this is, but if a host
> is running dq2 site services, then the dq2-clients package
> will not work - dq2 site services is the source of the
> /usr/lib/pythonXX/pycurl.so file... installed via an
> RPM package "python-curl-7.16.4-1". Here's a case where
> the VDT installer would perhaps benefit from better
> integration with the system RPM database, if it could detect
> that an incompatible pycurl.so were installed. Even nicer
> would be if somehow dq2-site-services and the dq2-clients
>
> But it's tricky, especially when things get installed via NFS.
> The server where this got installed did not have the dq2-site-services
> package installed, so even if the VDT installer tested for the
> incompatible pycurl.so, it would not have been found. It was only
> when this got NFS-mounted on a client that had the dq2-site-services
> RPMs installed that there was a problem. Not sure what, if anything,
> could have been done by the VDT installer to prevent this. It's a
> tricky problem.
>
> - Charles
>
>
>
Download (untitled) / with headers
text/plain 1.4k
Hi Charles,

I just want to make sure I'm following here. First you said, the problem
is solved, but then you said the following.

Also, I'm not sure what the dq2 packages contain and how they relate to
the VDT installation. I'm happy to insert extra checks if they make
sense, but I'm not sure which checks you are suggesting, and where they
should be.

Can you help me understand?

Thanks,
-alain

> Not sure how serious of a problem this is, but if a host
> is running dq2 site services, then the dq2-clients package
> will not work - dq2 site services is the source of the
> /usr/lib/pythonXX/pycurl.so file... installed via an
> RPM package "python-curl-7.16.4-1". Here's a case where
> the VDT installer would perhaps benefit from better
> integration with the system RPM database, if it could detect
> that an incompatible pycurl.so were installed. Even nicer
> would be if somehow dq2-site-services and the dq2-clients
>
> But it's tricky, especially when things get installed via NFS.
> The server where this got installed did not have the dq2-site-services
> package installed, so even if the VDT installer tested for the
> incompatible pycurl.so, it would not have been found. It was only
> when this got NFS-mounted on a client that had the dq2-site-services
> RPMs installed that there was a problem. Not sure what, if anything,
> could have been done by the VDT installer to prevent this. It's a
> tricky problem.
>
> - Charles
Download (untitled) / with headers
text/plain 780b
> In VDT curl distribution CURLOPT_CAINFO is set to:
> /home/condor/execute/dir_7315/userdir/curl/share/curl/curl-ca-bundle.crt

That is believable, but I can't find how it is set.

> Solution:
> compile curl in VDT with the correct path (or without CURLOPT_CAINFO)

What is "the correct path", since people can install the VDT into
different locations?

Unfortunately, I don't know how to do either of these. I think I
understand what you're explaining. If you have any deeper advice as to
how I can improve the distribution of curl in the VDT, I'm happy to
address it.

-alain

-----------------------------------------------------------------
Alain Roy vdt-support@opensciencegrid.org
VDT Support http://vdt.cs.wisc.edu/support.html
Subject: [vdt-support #3940] Problem with VDT and DQ2
Date: Sun, 14 Sep 2008 16:22:39 -0500
To: vdt-support@OPENSCIENCEGRID.ORG
From: Charles Waldman <cgw@hep.uchicago.edu>
Alain Roy via RT writes:
> Hi Charles,
>
> I just want to make sure I'm following here. First you said, the problem
> is solved, but then you said the following.

Well, the VDT package conflicts with something else that was installed
outside of the VDT framework, so I'm not sure if it's really an issue
for VDT to worry about. However, Marco chimed in with a suggestion
for how to avoid the collision, so it's probably a good idea to make
the changes Marco suggests.

Maybe chat on the phone Monday if there are still questions?

- Charles

>
> Also, I'm not sure what the dq2 packages contain and how they relate to
> the VDT installation. I'm happy to insert extra checks if they make
> sense, but I'm not sure which checks you are suggesting, and where they
> should be.
>
> Can you help me understand?
>
> Thanks,
> -alain
>
> > Not sure how serious of a problem this is, but if a host
> > is running dq2 site services, then the dq2-clients package
> > will not work - dq2 site services is the source of the
> > /usr/lib/pythonXX/pycurl.so file... installed via an
> > RPM package "python-curl-7.16.4-1". Here's a case where
> > the VDT installer would perhaps benefit from better
> > integration with the system RPM database, if it could detect
> > that an incompatible pycurl.so were installed. Even nicer
> > would be if somehow dq2-site-services and the dq2-clients
> >
> > But it's tricky, especially when things get installed via NFS.
> > The server where this got installed did not have the dq2-site-services
> > package installed, so even if the VDT installer tested for the
> > incompatible pycurl.so, it would not have been found. It was only
> > when this got NFS-mounted on a client that had the dq2-site-services
> > RPMs installed that there was a problem. Not sure what, if anything,
> > could have been done by the VDT installer to prevent this. It's a
> > tricky problem.
> >
> > - Charles
>
>
>
>
> --
> View ticket at <http://crt.cs.wisc.edu/Ticket/Display.html?user=guest&pass=guest&id=3940>
> VDT Support: vdt-support@opensciencegrid.org
Subject: Re: [vdt-support #3940] Problem with VDT and DQ2
Date: Sun, 14 Sep 2008 16:42:20 -0500
To: vdt-support@OPENSCIENCEGRID.ORG
From: Alain Roy <roy@cs.wisc.edu>
Download (untitled) / with headers
text/plain 910b
On Sep 14, 2008, at 4:25 PM, Charles G Waldman via RT wrote:
> Well, the VDT package conflicts with something else that was installed
> outside of the VDT framework, so I'm not sure if it's really an issue
> for VDT to worry about. However, Marco chimed in with a suggestion
> for how to avoid the collision, so it's probably a good idea to make
> the changes Marco suggests.
>
> Maybe chat on the phone Monday if there are still questions?

Got it.

I don't fully understand Marco's suggestion. If you can help me figure
out what to do, I'm happy to do it.

I'll be in meetings all day at Fermilab on Monday, but if we still
need a chat to figure things out on Tuesday, we can do it then.

Thanks,
-alain

-----------------------------------------------------------------
Alain Roy vdt-support@opensciencegrid.org
VDT Support http://vdt.cs.wisc.edu/support.html
CC: cgw@hep.uchicago.edu
Subject: Re: [vdt-support #3940] Problem with VDT and DQ2
Date: Sun, 14 Sep 2008 19:05:13 -0500 (CDT)
To: Alain Roy via RT <vdt-support@OPENSCIENCEGRID.ORG>
From: Marco Mambelli <marco@hep.uchicago.edu>
Download (untitled) / with headers
text/plain 847b
On Sun, 14 Sep 2008, Alain Roy via RT wrote:

>> In VDT curl distribution CURLOPT_CAINFO is set to:
>> /home/condor/execute/dir_7315/userdir/curl/share/curl/curl-ca-bundle.crt
>
> That is believable, but I can't find how it is set.
>
>> Solution:
>> compile curl in VDT with the correct path (or without CURLOPT_CAINFO)
>
> What is "the correct path", since people can install the VDT into
> different locations?
>
> Unfortunately, I don't know how to do either of these. I think I
> understand what you're explaining. If you have any deeper advice as to
> how I can improve the distribution of curl in the VDT, I'm happy to
> address it.
>

I'll check in curl compilation option.
If it is possible to use a variable,
VDT_LOCATION/curl/share/curl/curl-ca-bundle.crt would be the correct path
If not the value could be set to null.

Cheers,
Marco
CC: cgw@hep.uchicago.edu
Subject: Re: [vdt-support #3940] Problem with VDT and DQ2
Date: Sun, 14 Sep 2008 19:01:55 -0500 (CDT)
To: Alain Roy via RT <vdt-support@OPENSCIENCEGRID.ORG>
From: Marco Mambelli <marco@hep.uchicago.edu>
Download (untitled) / with headers
text/plain 1.7k
Hi Alain,
the problem is a variable set with a path existing only during the
compilation.
Curl has configuration variables that can be set with setopt(varname,
value) and affect the library behavior.
One of there is CAINFO that points to a file that contains certificates
considered trusted.
Somehow during compilation you set the default variable of this variable
to the file that is included in curl distribution but you put the full
path to the compilation directory (a subdirectory of condor home).
This directory does not exist in the pacman distribution, so the file is
not found.
A work around is to set the variable explicitly to some value.

Alain,
This week I'll be available for a IM chat or for a phone conversation.
Call me at 773 834 6696. It will be easier to explain.

Cheers,
Marco



On Sun, 14 Sep 2008, Alain Roy via RT wrote:

> On Sep 14, 2008, at 4:25 PM, Charles G Waldman via RT wrote:
>> Well, the VDT package conflicts with something else that was installed
>> outside of the VDT framework, so I'm not sure if it's really an issue
>> for VDT to worry about. However, Marco chimed in with a suggestion
>> for how to avoid the collision, so it's probably a good idea to make
>> the changes Marco suggests.
>>
>> Maybe chat on the phone Monday if there are still questions?
>
> Got it.
>
> I don't fully understand Marco's suggestion. If you can help me figure
> out what to do, I'm happy to do it.
>
> I'll be in meetings all day at Fermilab on Monday, but if we still
> need a chat to figure things out on Tuesday, we can do it then.
>
> Thanks,
> -alain
>
> -----------------------------------------------------------------
> Alain Roy vdt-support@opensciencegrid.org
> VDT Support http://vdt.cs.wisc.edu/support.html
>
>
>
Subject: Re: [vdt-support #3940] Problem with VDT and DQ2
Date: Sun, 14 Sep 2008 19:35:05 -0500
To: vdt-support@OPENSCIENCEGRID.ORG
From: Alain Roy <roy@cs.wisc.edu>
Download (untitled) / with headers
text/plain 519b
On Sep 14, 2008, at 7:05 PM, marco@hep.uchicago.edu via RT wrote:
> Somehow during compilation you set the default variable of this
> variable
> to the file that is included in curl distribution but you put the full
> path to the compilation directory (a subdirectory of condor home).

I'm not setting it explicitly, so it's based on configure --prefix
option somehow. (Because the prefix is /home/condor/execute/dir_...) I
couldn't find a way to pick a different default, but maybe I missed
something.

-alain
CC: cgw@hep.uchicago.edu
Subject: Re: [vdt-support #3940] Problem with VDT and DQ2
Date: Mon, 15 Sep 2008 14:01:38 -0500 (CDT)
To: Alain Roy via RT <vdt-support@OPENSCIENCEGRID.ORG>
From: Marco Mambelli <marco@hep.uchicago.edu>
Download (untitled) / with headers
text/plain 930b
Hi Alain,
I checked cUrl buils instructions and options (./configure -help)
I think adding "--without-ca-bundle" to your configure options may solve
the problem. It should disable the default CA file bundled with cUrl
--with-ca-bundle=FILE File name to use as CA bundle
--without-ca-bundle Don't use a default CA bundle

Cheers,
Marco

On Sun, 14 Sep 2008, Alain Roy via RT wrote:

> On Sep 14, 2008, at 7:05 PM, marco@hep.uchicago.edu via RT wrote:
>> Somehow during compilation you set the default variable of this
>> variable
>> to the file that is included in curl distribution but you put the full
>> path to the compilation directory (a subdirectory of condor home).
>
> I'm not setting it explicitly, so it's based on configure --prefix
> option somehow. (Because the prefix is /home/condor/execute/dir_...) I
> couldn't find a way to pick a different default, but maybe I missed
> something.
>
> -alain
>
>
>
Subject: Re: [vdt-support #3940] Problem with VDT and DQ2
Date: Tue, 16 Sep 2008 12:33:01 -0500
To: vdt-support@OPENSCIENCEGRID.ORG
From: Alain Roy <roy@cs.wisc.edu>
Download (untitled) / with headers
text/plain 1.2k
I saw that option, but I was worried that it would prevent https from
working at all. Do you know?

I'll give it a try.

-alain

On Sep 15, 2008, at 2:15 PM, marco@hep.uchicago.edu via RT wrote:
> http://crt.cs.wisc.edu/Ticket/Display.html?id=3940
>
> Hi Alain,
> I checked cUrl buils instructions and options (./configure -help)
> I think adding "--without-ca-bundle" to your configure options may
> solve
> the problem. It should disable the default CA file bundled with cUrl
> --with-ca-bundle=FILE File name to use as CA bundle
> --without-ca-bundle Don't use a default CA bundle
>
> Cheers,
> Marco
>
> On Sun, 14 Sep 2008, Alain Roy via RT wrote:
>
>> On Sep 14, 2008, at 7:05 PM, marco@hep.uchicago.edu via RT wrote:
>>> Somehow during compilation you set the default variable of this
>>> variable
>>> to the file that is included in curl distribution but you put the
>>> full
>>> path to the compilation directory (a subdirectory of condor home).
>>
>> I'm not setting it explicitly, so it's based on configure --prefix
>> option somehow. (Because the prefix is /home/condor/execute/
>> dir_...) I
>> couldn't find a way to pick a different default, but maybe I missed
>> something.
>>
>> -alain
>>
>>
>>
CC: cgw@hep.uchicago.edu
Subject: Re: [vdt-support #3940] Problem with VDT and DQ2
Date: Tue, 16 Sep 2008 13:11:27 -0500 (CDT)
To: Alain Roy via RT <vdt-support@OPENSCIENCEGRID.ORG>
From: Marco Mambelli <marco@hep.uchicago.edu>
Download (untitled) / with headers
text/plain 1.8k
Https will still work.
Https can use certificates from the bundle (containing standard
commercial CAs) and/or from a directory.
These paths can be set also at run time:
--cacert <CA certificate> (or env CURL_CA_BUNDLE)
--capath <CA certificate directory>
Then -k uses https ignoring CA certificates

When used in grid application usually --capath is set to the Globus CA dir

If you do a test after compilation without using any command line option
then you may have problems because the default cacert (that was set
before) is not set when using --without-ca-bundle

Thanks,
Marco



On Tue, 16 Sep 2008, Alain Roy via RT wrote:

> I saw that option, but I was worried that it would prevent https from
> working at all. Do you know?
>
> I'll give it a try.
>
> -alain
>
> On Sep 15, 2008, at 2:15 PM, marco@hep.uchicago.edu via RT wrote:
>> http://crt.cs.wisc.edu/Ticket/Display.html?id=3940
>>
>> Hi Alain,
>> I checked cUrl buils instructions and options (./configure -help)
>> I think adding "--without-ca-bundle" to your configure options may
>> solve
>> the problem. It should disable the default CA file bundled with cUrl
>> --with-ca-bundle=FILE File name to use as CA bundle
>> --without-ca-bundle Don't use a default CA bundle
>>
>> Cheers,
>> Marco
>>
>> On Sun, 14 Sep 2008, Alain Roy via RT wrote:
>>
>>> On Sep 14, 2008, at 7:05 PM, marco@hep.uchicago.edu via RT wrote:
>>>> Somehow during compilation you set the default variable of this
>>>> variable
>>>> to the file that is included in curl distribution but you put the
>>>> full
>>>> path to the compilation directory (a subdirectory of condor home).
>>>
>>> I'm not setting it explicitly, so it's based on configure --prefix
>>> option somehow. (Because the prefix is /home/condor/execute/
>>> dir_...) I
>>> couldn't find a way to pick a different default, but maybe I missed
>>> something.
>>>
>>> -alain
>>>
>>>
>>>
>
>
>
Download (untitled) / with headers
text/plain 1.6k
> Https will still work.
> Https can use certificates from the bundle (containing standard
> commercial CAs) and/or from a directory.
> These paths can be set also at run time:
> --cacert <CA certificate> (or env CURL_CA_BUNDLE)
> --capath <CA certificate directory>
> Then -k uses https ignoring CA certificates

I did some testing. I think the right thing to do is slightly different
than you suggest. See if you agree with me.

The VDT shouldn't ship the CA certificate bundle with Curl. The OS
vendors don't, but they configure curl to refer to the bundle that comes
with OpenSSL. Instead we should pass "--with-ca-bundle" and have it
point to the bundle that ships with OpenSSL. It's in different locations
for different OSes, but it's not too hard to figure out.

This way, the curl we ship will do the right thing by default, you can
still pass --capath to find the Globus CAs, and we won't have this
stupid hard-coded PATH in the VDT.

Does this sound like a good idea?

Even if it sounds like a good idea, I think it's not high priority
because you have things working now, so I'm not going to rush on this
unless you tell me it is high priority.

--------------------------

As an aside, the VDT isn't shipping the latest version of Curl, but the
latest version of Curl has a scary warning:

http://curl.haxx.se/lxr/source/lib/README.curl_off_t

Might this affect your usage of Curl from your Python scripts?

--------------------------

As another aside... Should the VDT be shipping curl at all? We're
shipping it because ATLAS requested it. Do you still care if it comes
from the VDT or the system?

-alain
Download (untitled) / with headers
text/plain 325b
On Debian 4, the CA bundle can be found in:
/etc/ssl/certs/ca-certificates.crt
This file is created when installing the ca-certificates package, even
though "dpkg -S" claims not to know about the file.

On RHAS 4, the CA bundle can be found in:
/usr/share/ssl/certs/ca-bundle.crt
This file is from the openssl RPM.
CC: cgw@hep.uchicago.edu
Subject: Re: [vdt-support #3940] Problem with VDT and DQ2
Date: Tue, 16 Sep 2008 14:31:48 -0500 (CDT)
To: Alain Roy via RT <vdt-support@OPENSCIENCEGRID.ORG>
From: Marco Mambelli <marco@hep.uchicago.edu>
Download (untitled) / with headers
text/plain 2.2k
On Tue, 16 Sep 2008, Alain Roy via RT wrote:
>> Https will still work.
>> Https can use certificates from the bundle (containing standard
>> commercial CAs) and/or from a directory.
>> These paths can be set also at run time:
>> --cacert <CA certificate> (or env CURL_CA_BUNDLE)
>> --capath <CA certificate directory>
>> Then -k uses https ignoring CA certificates
>
> I did some testing. I think the right thing to do is slightly different
> than you suggest. See if you agree with me.
>
> The VDT shouldn't ship the CA certificate bundle with Curl. The OS
> vendors don't, but they configure curl to refer to the bundle that comes
> with OpenSSL. Instead we should pass "--with-ca-bundle" and have it
> point to the bundle that ships with OpenSSL. It's in different locations
> for different OSes, but it's not too hard to figure out.
>
> This way, the curl we ship will do the right thing by default, you can
> still pass --capath to find the Globus CAs, and we won't have this
> stupid hard-coded PATH in the VDT.
>
> Does this sound like a good idea?

I like this idea, definitely better than mine.
Glad that you can find easily OpenSSL cert bundle.

> Even if it sounds like a good idea, I think it's not high priority
> because you have things working now, so I'm not going to rush on this
> unless you tell me it is high priority.

My fix is application specific (I modified the application to explicitly
unset the path to ca-bundle).
If someone else is using the library combination (curl+pycurl), it will
encounter the problem.
I think it is OK to roll it out with the next patch level (not need to
rush one out only for this)
Anyway I'm not really good/authoritative in judging priorities

> --------------------------
>
> As an aside, the VDT isn't shipping the latest version of Curl, but the
> latest version of Curl has a scary warning:
>
> http://curl.haxx.se/lxr/source/lib/README.curl_off_t
>
> Might this affect your usage of Curl from your Python scripts?

Current version in VDT is fine

> --------------------------
>
> As another aside... Should the VDT be shipping curl at all? We're
> shipping it because ATLAS requested it. Do you still care if it comes
> from the VDT or the system?

I know that most system provide it but some still don't and ATLAS pilots
rely on it.
I'd keep in in VDT

Thank you,
Marco
Subject: Re: [vdt-support #3940] Problem with VDT and DQ2
Date: Tue, 16 Sep 2008 14:51:52 -0500
To: vdt-support@OPENSCIENCEGRID.ORG
From: Alain Roy <roy@cs.wisc.edu>
Download (untitled) / with headers
text/plain 805b
Thanks for the feedback, Marco.

>>
>> As an aside, the VDT isn't shipping the latest version of Curl, but
>> the
>> latest version of Curl has a scary warning:
>>
>> http://curl.haxx.se/lxr/source/lib/README.curl_off_t
>>
>> Might this affect your usage of Curl from your Python scripts?
>
> Current version in VDT is fine

Can you clarify? I'm glad the current version is fine, but I see that
the new version has a ton of bug fixes. I like to update when it's
safe to update.

>> As another aside... Should the VDT be shipping curl at all? We're
>> shipping it because ATLAS requested it. Do you still care if it comes
>> from the VDT or the system?
>
> I know that most system provide it but some still don't and ATLAS
> pilots
> rely on it.
> I'd keep in in VDT

Good to know. Thanks!

-alain
CC: cgw@hep.uchicago.edu
Subject: Re: [vdt-support #3940] Problem with VDT and DQ2
Date: Tue, 16 Sep 2008 15:06:52 -0500 (CDT)
To: Alain Roy via RT <vdt-support@OPENSCIENCEGRID.ORG>
From: Marco Mambelli <marco@hep.uchicago.edu>
Download (untitled) / with headers
text/plain 912b
On Tue, 16 Sep 2008, Alain Roy via RT wrote:
> Thanks for the feedback, Marco.
>
>>>
>>> As an aside, the VDT isn't shipping the latest version of Curl, but
>>> the
>>> latest version of Curl has a scary warning:
>>>
>>> http://curl.haxx.se/lxr/source/lib/README.curl_off_t
>>>
>>> Might this affect your usage of Curl from your Python scripts?
>>
>> Current version in VDT is fine
>
> Can you clarify? I'm glad the current version is fine, but I see that
> the new version has a ton of bug fixes. I like to update when it's
> safe to update.

Off course. I checked the release notes and there are fixes and feature
improvements.
Anyway I intended to say that there is no known curl problem affecting
ATLAS and no request for any of the new features.

About libcurl use in ATLAS. As far as I know it is with curl binary and
with pycurl (if available); nothing else is linked to the library.

Thank you,
Marco
CC: MWT2 <mwt2-core-l@LISTSERV.INDIANA.EDU>
Subject: Broken curl
Date: Wed, 18 Feb 2009 10:25:55 -0500
To: vdt-support@OPENSCIENCEGRID.ORG
From: Sarah Williams <saewill@iupui.edu>
Download (untitled) / with headers
text/plain 2.6k
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dear VDT support,

At MWT2_UC and MWT2_IU, we appear to have a slightly broken version of
curl in our wn-client installation. It is pointed to a non-existent
location for the CAfile. If I run the command:
sarah@iut2-c002~$ curl -s --show-error --capath
/share/wn-client/globus/TRUSTED_CA --cert /tmp/x509up_u20037 --key
/tmp/x509up_u20037 https://gridui07.usatlas.bnl.gov:25443/server/panda/
curl: (77) error setting certificate verify locations:
CAfile:
/home/condor/execute/dir_7315/userdir/curl/share/curl/curl-ca-bundle.crt
CApath: /share/wn-client/globus/TRUSTED_CA

However, curl-config reports a different (and more correct) location:
sarah@iut2-c002/share/wn-client/curl/bin$ curl-config --ca
/share/wn-client/curl/share/curl/curl-ca-bundle.crt

Setting the CURL_CA_BUNDLE env variable does not fix the problem,
either. We've reverted to using the system curl temporarily.

Here is the update of vdt-version. Let me know if I can provide any
more information.

[root@uct2-grid7 wn-client]# vdt-version

You have installed a subset of VDT version 1.10.1k:

Software Status

- -------- ------

CGSI-gSOAP 1.2.1.2 OK

cURL 7.18.0 -

dccp (dCache client) 1.8.0-4 -

Fetch CRL 2.6.6 -

Globus Toolkit, pre web-services, client 4.0.7 -

Globus Toolkit, web-services, client 4.0.7 -

GPT 3.2autotools2004-NMI-9.0 -

Java 5 SDK 1.5.0_16 -

LCG-Utils 1.6.11-1p1 (includes GFAL 1.10.11-1) 1.6.11-1p1 -

LCG File Catalog Client 1.6.11.4 OK

Logrotate 3.7 -

MyProxy 4.2 -

Pegaus Worker Package 2.1.0 -

RLS, client 3.0.041021 -

SRM Fermi Client 1.8.0-15p8 -

SRM Berkeley Client 2.2.0.11.a -

UberFTP 2.0 UPDATE
AVAILABLE
vdt-update-certs 2.2 -

Wget 1.11

- --Sarah

- --
Sarah Williams
saewill@iupui.edu
(317) 274-0595
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x3E6F7BBA
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJnCiD2j+lnz5ve7oRAqNcAKCoMlXzVRKZtFOumRU25Z1guaHG5QCgtnP/
ds3rsvvfP3x/QtciuyZddCU=
=TqlM
-----END PGP SIGNATURE-----

 DQ2 was fixed with a workaround but now the Panda pilot is using the curl library as well and it is affected by the problem.

Marco

CC: vdt-support@OPENSCIENCEGRID.ORG, MWT2 <mwt2-core-l@LISTSERV.INDIANA.EDU>
Subject: Re: Broken curl
Date: Wed, 18 Feb 2009 11:16:06 -0500
To: Marco Mambelli <marco@hep.uchicago.edu>
From: Sarah Williams <saewill@iupui.edu>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Marco,

Thanks for pointing out the ticket and updating it!

- --Sarah

Marco Mambelli wrote:
> Hi Sarah,
> it is an old issue.
>
> see:
> http://crt.cs.wisc.edu/Ticket/Display.html?user=guest&pass=guest&id=3940
>
> Best,
> Marco
>
>
> On Wed, 18 Feb 2009, Sarah Williams wrote:
>
> Dear VDT support,
>
> At MWT2_UC and MWT2_IU, we appear to have a slightly broken version of
> curl in our wn-client installation. It is pointed to a non-existent
> location for the CAfile. If I run the command:
> sarah@iut2-c002~$ curl -s --show-error --capath
> /share/wn-client/globus/TRUSTED_CA --cert /tmp/x509up_u20037 --key
> /tmp/x509up_u20037 https://gridui07.usatlas.bnl.gov:25443/server/panda/
> curl: (77) error setting certificate verify locations:
> CAfile:
> /home/condor/execute/dir_7315/userdir/curl/share/curl/curl-ca-bundle.crt
> CApath: /share/wn-client/globus/TRUSTED_CA
>
> However, curl-config reports a different (and more correct) location:
> sarah@iut2-c002/share/wn-client/curl/bin$ curl-config --ca
> /share/wn-client/curl/share/curl/curl-ca-bundle.crt
>
> Setting the CURL_CA_BUNDLE env variable does not fix the problem,
> either. We've reverted to using the system curl temporarily.
>
> Here is the update of vdt-version. Let me know if I can provide any
> more information.
>
> [root@uct2-grid7 wn-client]# vdt-version
>
> You have installed a subset of VDT version 1.10.1k:
>
> Software Status
>
> -------- ------
>
> CGSI-gSOAP 1.2.1.2 OK
>
> cURL 7.18.0 -
>
> dccp (dCache client) 1.8.0-4 -
>
> Fetch CRL 2.6.6 -
>
> Globus Toolkit, pre web-services, client 4.0.7 -
>
> Globus Toolkit, web-services, client 4.0.7 -
>
> GPT 3.2autotools2004-NMI-9.0 -
>
> Java 5 SDK 1.5.0_16 -
>
> LCG-Utils 1.6.11-1p1 (includes GFAL 1.10.11-1) 1.6.11-1p1 -
>
> LCG File Catalog Client 1.6.11.4 OK
>
> Logrotate 3.7 -
>
> MyProxy 4.2 -
>
> Pegaus Worker Package 2.1.0 -
>
> RLS, client 3.0.041021 -
>
> SRM Fermi Client 1.8.0-15p8 -
>
> SRM Berkeley Client 2.2.0.11.a -
>
> UberFTP 2.0 UPDATE
> AVAILABLE
> vdt-update-certs 2.2 -
>
> Wget 1.11
>
> --Sarah
>
>>

- --
Sarah Williams
saewill@iupui.edu
(317) 274-0595
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x3E6F7BBA
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJnDRG2j+lnz5ve7oRApTcAJoDmpvF4QktCXf4wOdok4EaprIu/gCfWvFu
r3WWc+SHiY33dstskH52AGA=
=7Qjj
-----END PGP SIGNATURE-----

 Adding the option '--cacert /share/wn-client/curl/share/curl/curl-ca-bundle.crt' or setting the environment variable CURL_CA_BUNDLE=/share/wn-client/curl/share/curl/curl-ca-bundle.crt will override the value wrongly set during VDT compilation and fix the shell invocation of curl.

I recommend to bump the priority of this problem and at least add the option not to include any certificate bundle file. Workaround exist but the problem has been recurring many times.

This would fix all the problems encountered so far (grid programs use their own certificates in capath)

I understand that pointing to the system ca bundle would be better but setting that correctly may require some effort (file discovery) while pointing to no file is very easy (compilation option).

Thank you,

Marco

A correction to the previously pointed workaround (was: Adding the option '--cacert /share/wn-client/curl/share/curl/curl-ca-bundle.crt' or setting the environment variable CURL_CA_BUNDLE=/share/wn-client/curl/share/curl/curl-ca-bundle.crt will...)

That path was specific to one site, on all OSG sites the file path is $OSG_GRID/curl/share/curl/curl-ca-bundle.crt (on the specific site it was OSG_GRID=share/wn-client).

After sourcing $OSG_GRID/setup.sh (or the setup of your generic VDT installation if you are not using WN-client in VDT) you can use also $VDT_LOCATION/curl/share/curl/curl-ca-bundle.crt

Marco

Download (untitled) / with headers
text/plain 723b
Hi Sarah,

Thanks for prodding us on this issue. As Marco pointed out, he's
reported it before. Coincidentally he's been having this problem rear
its ugly head again. Earlier, it was easy for us to work around it, but
it's getting harder, so I will elevate the priority of this work and
will get it done soon.

I'm going to merge your ticket with his, so it's easy to update both of
you at the same time. It will become ticket #3940.
http://crt.cs.wisc.edu/Ticket/Display.html?user=guest&pass=guest&id=3940

Thanks,
-alain
-----------------------------------------------------------------
Alain Roy vdt-support@opensciencegrid.org
VDT Support http://vdt.cs.wisc.edu/support.html
Download (untitled) / with headers
text/plain 1.3k
Hi Marco,

Earlier I suggested pointing curl at the system version of the
curl-ca-bundle, which ships with SSL. Is that still acceptable to you?
This way we get updates from the OS vendor instead of waiting for the
VDT to ship them.

The other possibility would be to make the VDT/OSG certificates
available as a CA bundles. This would be a disjoint set of CA
certificates. This adds some complexity to the solution, so unless we
need that I would stick with the system bundle.

What do you think?

-alain

-----------------------------------------------------------------
Alain Roy vdt-support@opensciencegrid.org
VDT Support http://vdt.cs.wisc.edu/support.html

On Wed Feb 18 10:52:08 2009, guest wrote:
> A correction to the previously pointed workaround (was: Adding the
> option
> '--cacert /share/wn-client/curl/share/curl/curl-ca-bundle.crt' or
> setting the
> environment variable
> CURL_CA_BUNDLE=/share/wn-client/curl/share/curl/curl-ca-bundle.crt
> will...)
>
> That path was specific to one site, on all OSG sites the file path is
> $OSG_GRID/curl/share/curl/curl-ca-bundle.crt (on the specific site it
> was
> OSG_GRID=share/wn-client).
>
> After sourcing $OSG_GRID/setup.sh (or the setup of your generic VDT
> installation if you are not using WN-client in VDT) you can use also
> $VDT_LOCATION/curl/share/curl/curl-ca-bundle.crt
>
> Marco
CC: cgw@hep.uchicago.edu, saewill@iupui.edu
Subject: Re: [vdt-support #3940] Fix how Curl finds the CA bundle
Date: Wed, 18 Feb 2009 15:29:10 -0600 (CST)
To: Alain Roy via RT <vdt-support@OPENSCIENCEGRID.ORG>
From: Marco Mambelli <marco@hep.uchicago.edu>
Download (untitled) / with headers
text/plain 2.4k
Hi Alain,
yes.

Pointing curl at the system version of the curl-ca-bundle, which ships
with SSL is the best solution (it i consistent with other SSL
applications on the system).

If that takes effort also having no curl-ca-bundle is fine (this should
be pertty easy: change one string with a fixed value and recompile).
It the grid worls (OSG, but also othe Globus based grids) the important
part with all the used certificates is the capath that is setup
explicitely.

I don't like the other solution (VDT/OSG distributing the CA bundle) and I
don't understand its utility:
- would this contain the Grid CAs (already in capath)?
No if it is a disjoint set
- would this contain the 'commercial' CAs normally in SSL?
1. OSG users and services are not using them (as far as I know).
2. Better to use the one that is already in every system, also
OpenSSL is not in VDT (and I'm glad it is not there - I'm not considering
some library files in Globus)

Best,
Marco


On Wed, 18 Feb 2009, Alain Roy via RT wrote:

> Hi Marco,
>
> Earlier I suggested pointing curl at the system version of the
> curl-ca-bundle, which ships with SSL. Is that still acceptable to you?
> This way we get updates from the OS vendor instead of waiting for the
> VDT to ship them.
>
> The other possibility would be to make the VDT/OSG certificates
> available as a CA bundles. This would be a disjoint set of CA
> certificates. This adds some complexity to the solution, so unless we
> need that I would stick with the system bundle.
>
> What do you think?
>
> -alain
>
> -----------------------------------------------------------------
> Alain Roy vdt-support@opensciencegrid.org
> VDT Support http://vdt.cs.wisc.edu/support.html
>
> On Wed Feb 18 10:52:08 2009, guest wrote:
>> A correction to the previously pointed workaround (was: Adding the
>> option
>> '--cacert /share/wn-client/curl/share/curl/curl-ca-bundle.crt' or
>> setting the
>> environment variable
>> CURL_CA_BUNDLE=/share/wn-client/curl/share/curl/curl-ca-bundle.crt
>> will...)
>>
>> That path was specific to one site, on all OSG sites the file path is
>> $OSG_GRID/curl/share/curl/curl-ca-bundle.crt (on the specific site it
>> was
>> OSG_GRID=share/wn-client).
>>
>> After sourcing $OSG_GRID/setup.sh (or the setup of your generic VDT
>> installation if you are not using WN-client in VDT) you can use also
>> $VDT_LOCATION/curl/share/curl/curl-ca-bundle.crt
>>
>> Marco
>
>
>
>
>
Subject: Re: [vdt-support #3940] Fix how Curl finds the CA bundle
Date: Wed, 18 Feb 2009 15:49:04 -0600
To: vdt-support@OPENSCIENCEGRID.ORG
From: Alain Roy <roy@cs.wisc.edu>
Download (untitled) / with headers
text/plain 1.1k
Good, then I think we agree. We'll get this done soon.

Thanks,
-alain

On Feb 18, 2009, at 3:29 PM, marco@hep.uchicago.edu via RT wrote:
> Pointing curl at the system version of the curl-ca-bundle, which ships
> with SSL is the best solution (it i consistent with other SSL
> applications on the system).
>
> If that takes effort also having no curl-ca-bundle is fine (this
> should
> be pertty easy: change one string with a fixed value and recompile).
> It the grid worls (OSG, but also othe Globus based grids) the
> important
> part with all the used certificates is the capath that is setup
> explicitely.
>
> I don't like the other solution (VDT/OSG distributing the CA bundle)
> and I
> don't understand its utility:
> - would this contain the Grid CAs (already in capath)?
> No if it is a disjoint set
> - would this contain the 'commercial' CAs normally in SSL?
> 1. OSG users and services are not using them (as far as I know).
> 2. Better to use the one that is already in every system, also
> OpenSSL is not in VDT (and I'm glad it is not there - I'm not
> considering
> some library files in Globus)
>
> Best,
> Marco
Subject: [vdt-support #3940] SVN commit, rev 8748
To: vdt-support@cs.wisc.edu
From: cat@cs.wisc.edu
Download (untitled) / with headers
text/plain 176b
Commit comment:
Making a branch for Leo to use.


Changed files:
A vdt/branches/vdt-1.10.1-leo-curl/

To generate a diff:
svn diff -c 8748 file:///p/condor/workspaces/vdt/svn
Subject: [vdt-support #3940] SVN commit, rev 8749
To: vdt-support@cs.wisc.edu
From: cat@cs.wisc.edu
Download (untitled) / with headers
text/plain 175b
Commit comment:
Fixed VDT_VERSION in defs.


Changed files:
U vdt/branches/vdt-1.10.1-leo-curl/defs

To generate a diff:
svn diff -c 8749 file:///p/condor/workspaces/vdt/svn
Subject: [vdt-support #3940] SVN commit, rev 8774
To: vdt-support@cs.wisc.edu
From: cat@cs.wisc.edu
Download (untitled) / with headers
text/plain 227b
Commit comment:
Tweaks to Leo's code for clarity.


Changed files:
U vdt/branches/vdt-1.10.1-leo-curl/VDT-Certification-Tests/vdt/tests/tests/install.t

To generate a diff:
svn diff -c 8774 file:///p/condor/workspaces/vdt/svn
Subject: [vdt-support #3940] SVN commit, rev 8776
To: vdt-support@cs.wisc.edu
From: cat@cs.wisc.edu
Download (untitled) / with headers
text/plain 348b
Commit comment:
Tweaked the new '/home/condor' test -- Scot usefully pointed out that there
should be only one test for all executables, which actually simplified things.


Changed files:
U vdt/branches/vdt-1.10.1-leo-curl/VDT-Certification-Tests/vdt/tests/tests/install.t

To generate a diff:
svn diff -c 8776 file:///p/condor/workspaces/vdt/svn
Subject: [vdt-support #3940] SVN commit, rev 8777
To: vdt-support@cs.wisc.edu
From: cat@cs.wisc.edu
Download (untitled) / with headers
text/plain 169b
Commit comment:
Tag before merge.


Changed files:
A vdt/branches/vdt-1.10.1-leo-curl-merge1/

To generate a diff:
svn diff -c 8777 file:///p/condor/workspaces/vdt/svn
Subject: [vdt-support #3940] SVN commit, rev 8779
To: vdt-support@cs.wisc.edu
From: cat@cs.wisc.edu
Download (untitled) / with headers
text/plain 587b
Commit comment:
Merge from vdt-1.10.1-leo-curl branch:

svn merge -r 8748:8778 $SVN/vdt/branches/vdt-1.10.1-leo-curl .

Revision 8777 = tags/vdt-1.10.1-leo-curl-merge1

Rebuilt cURL to eliminate '/home/condor' from its binaries. Added a new
install.t test to make sure that certain (fixed) binaries do not include
'/home/condor'.


Changed files:
U vdt/branches/vdt-1.10.1/Curl/nmi/build-curl.pl
U vdt/branches/vdt-1.10.1/VDT-Certification-Tests/vdt/tests/tests/install.t
U vdt/branches/vdt-1.10.1/defs

To generate a diff:
svn diff -c 8779 file:///p/condor/workspaces/vdt/svn
Subject: [vdt-support #3940] SVN commit, rev 8781
To: vdt-support@cs.wisc.edu
From: cat@cs.wisc.edu
Download (untitled) / with headers
text/plain 207b
Commit comment:
Added 1.10.99 back to testing for the pending cURL update.


Changed files:
U tests/trunk/test-scripts/tests-to-run

To generate a diff:
svn diff -c 8781 file:///p/condor/workspaces/vdt/svn
Subject: [vdt-support #3940] cURL update
Date: Wed, 04 Mar 2009 11:55:29 -0600
To: vdt-support@OPENSCIENCEGRID.ORG
From: Tim Cartwright <cat@cs.wisc.edu>
Download (untitled) / with headers
text/plain 536b
Marco (et al.):

There was a lot of history to this ticket regarding cURL and its VDT-
related
build issues. Your most recent summary of the situation was this:

> Pointing curl at the system version of the curl-ca-bundle, which
ships with
> SSL is the best solution (it i consistent with other SSL
applications on the
> system).

We released VDT 1.10.1u today, and it fixed this problem and the one
about
referring to /home/condor. The release notes are here:

http://vdt.cs.wisc.edu/releases/1.10.1/release-u.html

-- Tim