[PyGreSQL] [patch] relinquish GIL while connecting to database

[Peter Schuller]

Hello,

> > It uses krb5 in a non-threadsafe fashion (such as in fe-auth.c). In
> > modern versions of libpq there is additional locking, and there are
> > comments specifically mentioning that it is needed for krb5.
> > 
> > The end-result is sporadic segfaults.
> 
> But this happens only when you have rules with krb4/krb5 active in your 
> pg_hba.conf, right?

No. This happens long before connectivity with the server has been
established. Essentially:

PQsetdblogin() internally creates a connection info object from an
empty string. This leads to code further down (I can check again which
functions exactly if you wish, but I don't remember off hand) trying
to perform an automatic default username discovery. This is done via
kerberos. Here is a sample stacktrace from a crash when the stack
wasn't borked:

#0  0xb7c5c1a2 in krb5_os_free_context () from /usr/lib/libkrb5.so.3
#1  0xb7c4725a in krb5_free_context () from /usr/lib/libkrb5.so.3
#2  0xb7da2c5f in fe_getauthname () from /usr/lib/libpq.so.3
#3  0xb7da2d0f in PQconnectdb () from /usr/lib/libpq.so.3
#4  0xb7da35eb in PQsetdbLogin () from /usr/lib/libpq.so.3
#5  0xb7da5cad in PQsetNoticeProcessor () from /usr/lib/libpq.so.3
#6  0xb7da36dc in PQsetdbLogin () from /usr/lib/libpq.so.3
#7  0xb7da3b5a in PQconnectPoll () from /usr/lib/libpq.so.3
#8  0xb7dcd72b in init_pg () from /usr/lib/python2.3/site-packages/_pg.so

fe_getauthname() being the function that had locking added (which may
also be true of other places; I didn't check).

The krb5 lookup is however predicated on some globals, whose
setting/not setting I haven't investigated. My workaround is instead
to just built without krb5. But the point being, this happens even
without use of kerberos anywhere.

> > I have not investigated exactly as of which version of libpq this
> > should be fixed, but the sarge version is 7.4.7.
> 
> As far as I see, the precautions you mentioned have first been included 
> in PG 8.0. So the problems should only appear when you're using native 
> Kerberos with PG < 8.0, which is a quite deprecated configuration.

Suspected that would be the case. And yes, quite deprecated indeed. I
imagine any distribution/OS with a sufficiently new version of pgdb
would have a much newer libpq than 7.x - even if some people may be on
7.x servers for various reasons. People installing custom versions by
source, mixing old/new stuff, are likely to check the ChangeLog I
hope.

> Maybe we should wrap the Py_BEGIN_ALLOW_THREADS in something like
> #if PG_VERSION < 8.0 && (defined(krb4) || defined(krb5))
> Though this checks only the pq lib against which PyGreSQL is built, not 
> the one that is actually running on the system. But at least if you then 
> build PyGreSQL on your old Debian system, things should work.

On the other hand you have to jump through some hoops to run with a
7.x when it was built with 8.x, so that feels like a pretty useful
check.

Btw, I also noticed that the calls to PQfinish() are subject to the
same problem. One of the calls I fixed and tested and it seems stable
(the one on explicit close()). I didn't test relinquing the GIL in the
case of the object being garbage collected; I'm not sure if there is
any special requirement with respect to de-allocation that would
preclue relinquishing the GIL.

I could submit a patch but on the other hand it's probably more work
to apply a patch than just go in and modify the source.

Any thoughts? Would be nice to remove all forms of interpreter
blocking if possible. Does anyone here feel comfortable
w.r.t. releasing the GIL during de-allocation?

-- 
/ Peter Schuller

PGP userID: 0xE9758B7D or 'Peter Schuller <peter.schuller at infidyne.com>'
Key retrieval: Send an E-Mail to getpgpkey at scode.org
E-Mail: peter.schuller at infidyne.com Web: http://www.scode.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
Url : http://mailman.vex.net/pipermail/pygresql/attachments/20080922/075b5e51/attachment.pgp