From mboxrd@z Thu Jan 1 00:00:00 1970 Delivered-To: chneukirchen@gmail.com Received: by 10.229.109.203 with SMTP id k11cs16261qcp; Thu, 4 Aug 2011 16:26:21 -0700 (PDT) Return-Path: Received-SPF: pass (google.com: domain of rack-devel+bncCP_V2_zRBRCY1ezxBBoELS_9-Q@googlegroups.com designates 10.236.165.106 as permitted sender) client-ip=10.236.165.106; Authentication-Results: mr.google.com; spf=pass (google.com: domain of rack-devel+bncCP_V2_zRBRCY1ezxBBoELS_9-Q@googlegroups.com designates 10.236.165.106 as permitted sender) smtp.mail=rack-devel+bncCP_V2_zRBRCY1ezxBBoELS_9-Q@googlegroups.com; dkim=pass header.i=rack-devel+bncCP_V2_zRBRCY1ezxBBoELS_9-Q@googlegroups.com Received: from mr.google.com ([10.236.165.106]) by 10.236.165.106 with SMTP id d70mr772636yhl.45.1312500379243 (num_hops = 1); Thu, 04 Aug 2011 16:26:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=beta; h=x-beenthere:received-spf:from:mime-version:subject:date:in-reply-to :to:references:message-id:x-mailer:x-original-sender :x-original-authentication-results:reply-to:precedence:mailing-list :list-id:x-google-group-id:list-post:list-help:list-archive:sender :list-subscribe:list-unsubscribe:content-type; bh=s2hWRyHt2/weOMeXrvyqvFOMR+1YyuQrGkkHHyuN5mE=; b=QfRCPxpzFwIaxPgcBglrdWZfm2g8Le8zZK4cvlafeeIQAYRBT9pO4s5KJBKdtqjPUA /csrkAKzXAusmXQwwNKhUsB+m75g7p6SgQNaUmXRPRurZWwhCv2iPH3lsSRihuqF0Pvh i5CTJEZ+y2QSa1Bj7cx35TV8z/5R5CSqMFBaU= Received: by 10.236.165.106 with SMTP id d70mr237305yhl.45.1312500376893; Thu, 04 Aug 2011 16:26:16 -0700 (PDT) X-BeenThere: rack-devel@googlegroups.com Received: by 10.231.77.5 with SMTP id e5ls2512556ibk.0.gmail; Thu, 04 Aug 2011 16:26:15 -0700 (PDT) Received: by 10.42.131.68 with SMTP id y4mr813298ics.20.1312500375491; Thu, 04 Aug 2011 16:26:15 -0700 (PDT) Received: by 10.42.131.68 with SMTP id y4mr813297ics.20.1312500375474; Thu, 04 Aug 2011 16:26:15 -0700 (PDT) Received: from mail-iy0-f178.google.com (mail-iy0-f178.google.com [209.85.210.178]) by gmr-mx.google.com with ESMTPS id gh9si2315025icb.3.2011.08.04.16.26.15 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 04 Aug 2011 16:26:15 -0700 (PDT) Received-SPF: pass (google.com: domain of jftucker@gmail.com designates 209.85.210.178 as permitted sender) client-ip=209.85.210.178; Received: by iym5 with SMTP id 5so2336285iym.37 for ; Thu, 04 Aug 2011 16:26:15 -0700 (PDT) Received: by 10.42.130.137 with SMTP id v9mr1189217ics.395.1312500375305; Thu, 04 Aug 2011 16:26:15 -0700 (PDT) Received: from [192.168.2.56] (216-75-227-188.static.wiline.com [216.75.227.188]) by mx.google.com with ESMTPS id f14sm3188048icm.15.2011.08.04.16.26.12 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 04 Aug 2011 16:26:13 -0700 (PDT) From: James Tucker Mime-Version: 1.0 (Apple Message framework v1244.3) Subject: Re: Session collisions on rails 3.1rc4 (authlogic, omniauth, memcache store, passenger) Date: Thu, 4 Aug 2011 16:26:11 -0700 In-Reply-To: <2CD439944A9549FA982EC7D0A8A85EA0@gmail.com> To: rack-devel@googlegroups.com References: <8F93F290D08845C89C2A7C1519A9138B@gmail.com> <4E31BE74.8060300@gmail.com> <2CD439944A9549FA982EC7D0A8A85EA0@gmail.com> Message-Id: X-Mailer: Apple Mail (2.1244.3) X-Original-Sender: jftucker@gmail.com X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jftucker@gmail.com designates 209.85.210.178 as permitted sender) smtp.mail=jftucker@gmail.com; dkim=pass (test mode) header.i=@gmail.com Reply-To: rack-devel@googlegroups.com Precedence: list Mailing-list: list rack-devel@googlegroups.com; contact rack-devel+owners@googlegroups.com List-ID: X-Google-Group-Id: 486215384060 List-Post: , List-Help: , List-Archive: Sender: rack-devel@googlegroups.com List-Subscribe: , List-Unsubscribe: , Content-Type: multipart/alternative; boundary="Apple-Mail=_4A7E97C7-A5F2-4AE8-94D9-1EC21B8CF504" --Apple-Mail=_4A7E97C7-A5F2-4AE8-94D9-1EC21B8CF504 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Maybe once I've had time to do a proper review. On Aug 3, 2011, at 3:32 AM, Joshua Ballanco wrote: > Hi Neil, >=20 > Good to hear you figured that out. I, too, had heard mention of Dalli = as a replacement for memcache-client. Perhaps we should update Rack's = gemspec? >=20 > - Josh > On Thursday, July 28, 2011 at 10:54 PM, Neil Matatall wrote: >=20 >> Joshua, >>=20 >> I think what you described was indeed our issue. The call to = @pool.add >> actually returned "END" which is a sign of a get_multikey >>=20 >> We were using memcache-client and the author informed me that the >> project was no longer supported and said that this can occur when the >> same memcache instance handles sessions and data. He recommended >> splitting the caches as well as switching to Dalli instead. >>=20 >> So...not a rack issue! >>=20 >> Thanks! >> Neil >>=20 >> On 7/25/11 1:23 PM, Joshua Ballanco wrote: >>> Have you considered the possibility that memcache might be recycling >>> keys on you? >>>=20 >>> We had an issue a while back where we were using memcache for both >>> fragment caching and session storage. Occasionally, we would get >>> exceptions in retrieving a session, and looking at the error message >>> it was clear that what was retrieved from memcache was a fragment = and >>> not a session. Unfortunately, we moved back to a cookie-based = session >>> store before we had a chance to look deeper into the issue. What I = can >>> say is that based on the key generation scheme we were using for >>> sessions and cache keys, there was effectively 0 chance that we were >>> duplicating keys. Instead, it seemed like memcache was re-using = slots >>> for different keys when we started exhausting free slots. >>>=20 >>> Hope that helps! >>>=20 >>> - Joshua Ballanco >>>=20 >>> On Monday, July 25, 2011 at 2:22 PM, Neil wrote: >>>=20 >>>> While it's entirely possible that this issue is caused by some = other >>>> factor, but we are getting session collisions as well as an issue >>>> where one user is getting another user's session. This is clearly >>>> bad, but I cannot for the life of me figure out how this could even >>>> happen in the first place. The code looks thread safe to me, and a >>>> quick discussion on #ruby-lang seems to support that. >>>>=20 >>>> Thoughts: >>>> 1. Session IDs are being generated in the same sequence (uses >>>> securerandom -> openssl which does not have a static seed) >>>> 2. Threads. Looks good to me. >>>> 3. Maybe memcached is returning something other than "STORED/ >>>> NOT_STORED" for @pool.add(sid, session), but the operation still >>>> succeeded? >>>> 4. Gnomes. >>>>=20 >>>> Any input is GREATLY appreciated. Please don't say "it's an RC, = what >>>> do you expect?" :) >>>>=20 >>>>=20 >>>> From >>>> = https://github.com/rack/rack/blob/master/lib/rack/session/memcache.rb >>>> def generate_sid >>>> loop do >>>> sid =3D super >>>> break sid unless @pool.get(sid, true) >>>> end >>>> end >>>>=20 >>>> def get_session(env, sid) >>>> with_lock(env, [nil, {}]) do >>>> unless sid and session =3D @pool.get(sid) >>>> sid, session =3D generate_sid, {} >>>> unless /^STORED/ =3D~ @pool.add(sid, session) >>>> raise "Session collision on '#{sid.inspect}'" >>>> end >>>> end >>>> [sid, session] >>>> end >>>> end >>>>=20 >>>> def set_session(env, session_id, new_session, options) >>>> expiry =3D options[:expire_after] >>>> expiry =3D expiry.nil? ? 0 : expiry + 1 >>>>=20 >>>> with_lock(env, false) do >>>> @pool.set session_id, new_session, expiry >>>> session_id >>>> end >>>> end >=20 --Apple-Mail=_4A7E97C7-A5F2-4AE8-94D9-1EC21B8CF504 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii Maybe = once I've had time to do a proper review.

On Aug 3, = 2011, at 3:32 AM, Joshua Ballanco wrote:

Hi Neil,

Good to hear you = figured that out. I, too, had heard mention of Dalli as a replacement = for memcache-client. Perhaps we should update Rack's = gemspec?

- Josh

On Thursday, = July 28, 2011 at 10:54 PM, Neil Matatall wrote:

Joshua,

I think what you = described was indeed our issue. The call to @pool.add
actually = returned "END" which is a sign of a get_multikey

We were using = memcache-client and the author informed me that the
project was no = longer supported and said that this can occur when the
same memcache = instance handles sessions and data. He recommended
splitting the = caches as well as switching to Dalli instead.

So...not a rack = issue!

Thanks!
Neil

On 7/25/11 1:23 PM, Joshua Ballanco = wrote:
Have you considered the = possibility that memcache might be recycling
keys on you?

We = had an issue a while back where we were using memcache for = both
fragment caching and session storage. Occasionally, we would = get
exceptions in retrieving a session, and looking at the error = message
it was clear that what was retrieved from memcache was a = fragment and
not a session. Unfortunately, we moved back to a = cookie-based session
store before we had a chance to look deeper into = the issue. What I can
say is that based on the key generation scheme = we were using for
sessions and cache keys, there was effectively 0 = chance that we were
duplicating keys. Instead, it seemed like = memcache was re-using slots
for different keys when we started = exhausting free slots.

Hope that helps!

- Joshua = Ballanco

On Monday, July 25, 2011 at 2:22 PM, Neil = wrote:

While it's entirely = possible that this issue is caused by some other
factor, but we are = getting session collisions as well as an issue
where one user is = getting another user's session. This is clearly
bad, but I cannot for = the life of me figure out how this could even
happen in the first = place. The code looks thread safe to me, and a
quick discussion on = #ruby-lang seems to support that.

Thoughts:
1. Session IDs are = being generated in the same sequence (uses
securerandom -> openssl = which does not have a static seed)
2. Threads. Looks good to = me.
3. Maybe memcached is returning something other than = "STORED/
NOT_STORED" for @pool.add(sid, session), but the operation = still
succeeded?
4. Gnomes.

Any input is GREATLY = appreciated. Please don't say "it's an RC, what
do you expect?" = :)


From
https://github.com/rack/rack/blob/master/lib/rack/session/memcache.rb=
def generate_sid
loop do
sid =3D super
break sid unless = @pool.get(sid, true)
end
end

def get_session(env, = sid)
with_lock(env, [nil, {}]) do
unless sid and session =3D = @pool.get(sid)
sid, session =3D generate_sid, {}
unless /^STORED/ = =3D~ @pool.add(sid, session)
raise "Session collision on = '#{sid.inspect}'"
end
end
[sid, = session]
end
end

def set_session(env, session_id, = new_session, options)
expiry =3D options[:expire_after]
expiry =3D = expiry.nil? ? 0 : expiry + 1

with_lock(env, false) = do
@pool.set session_id, new_session, = expiry
session_id
end
end
 =20 =20 =20 =20
=20


= --Apple-Mail=_4A7E97C7-A5F2-4AE8-94D9-1EC21B8CF504--