From mboxrd@z Thu Jan 1 00:00:00 1970 Delivered-To: chneukirchen@gmail.com Received: by 10.229.70.138 with SMTP id d10cs21071qcj; Mon, 25 Jul 2011 14:12:56 -0700 (PDT) Return-Path: Received-SPF: pass (google.com: domain of rack-devel+bncCOiF2Lu6BRDVuLfxBBoEtM8xTA@googlegroups.com designates 10.236.185.97 as permitted sender) client-ip=10.236.185.97; Authentication-Results: mr.google.com; spf=pass (google.com: domain of rack-devel+bncCOiF2Lu6BRDVuLfxBBoEtM8xTA@googlegroups.com designates 10.236.185.97 as permitted sender) smtp.mail=rack-devel+bncCOiF2Lu6BRDVuLfxBBoEtM8xTA@googlegroups.com; dkim=pass header.i=rack-devel+bncCOiF2Lu6BRDVuLfxBBoEtM8xTA@googlegroups.com Received: from mr.google.com ([10.236.185.97]) by 10.236.185.97 with SMTP id t61mr1886807yhm.59.1311628375502 (num_hops = 1); Mon, 25 Jul 2011 14:12:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=beta; h=x-beenthere:received-spf:from:mime-version:subject:date:in-reply-to :to:references:message-id:x-mailer:x-original-sender :x-original-authentication-results:reply-to:precedence:mailing-list :list-id:x-google-group-id:list-post:list-help:list-archive:sender :list-subscribe:list-unsubscribe:content-type; bh=yJzK+JTcfhwKNQKVe44Vae/i6PK+O+yv8Jl6+m/6rc8=; b=3KRFsRqWjEROQ/JBFilqMsnY414MOG/jYv5JS61ZXWUQhT9dNW8iPWgooitlmOyvx9 +EH+pG8XcGOQX+epMVTfnvjhiJDl4xFeOtHN9fSITTgI002l+rEhOBOasOChqDcqMfSS DKvIwd6q7BynHaM+Y+3J3UUHnC4ugL73+PrXg= Received: by 10.236.185.97 with SMTP id t61mr540827yhm.59.1311628373367; Mon, 25 Jul 2011 14:12:53 -0700 (PDT) X-BeenThere: rack-devel@googlegroups.com Received: by 10.100.30.24 with SMTP id d24ls2643821and.6.gmail; Mon, 25 Jul 2011 14:12:51 -0700 (PDT) Received: by 10.236.9.101 with SMTP id 65mr2409267yhs.173.1311628371142; Mon, 25 Jul 2011 14:12:51 -0700 (PDT) Received: by 10.236.9.101 with SMTP id 65mr2409266yhs.173.1311628371118; Mon, 25 Jul 2011 14:12:51 -0700 (PDT) Received: from mail-gw0-f49.google.com (mail-gw0-f49.google.com [74.125.83.49]) by gmr-mx.google.com with ESMTPS id p47si2149394yhl.2.2011.07.25.14.12.51 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 25 Jul 2011 14:12:51 -0700 (PDT) Received-SPF: pass (google.com: domain of neil.matatall@gmail.com designates 74.125.83.49 as permitted sender) client-ip=74.125.83.49; Received: by gwb1 with SMTP id 1so4061296gwb.22 for ; Mon, 25 Jul 2011 14:12:51 -0700 (PDT) Received: by 10.68.8.38 with SMTP id o6mr2168263pba.129.1311628370880; Mon, 25 Jul 2011 14:12:50 -0700 (PDT) Received: from [192.168.0.103] (67.110.253.162.ptr.us.xo.net [67.110.253.162]) by mx.google.com with ESMTPS id g4sm4924289pbj.57.2011.07.25.14.12.48 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 25 Jul 2011 14:12:48 -0700 (PDT) From: Neil Matatall Mime-Version: 1.0 (Apple Message framework v1244.3) Subject: Re: Session collisions on rails 3.1rc4 (authlogic, omniauth, memcache store, passenger) Date: Mon, 25 Jul 2011 14:12:47 -0700 In-Reply-To: <8F93F290D08845C89C2A7C1519A9138B@gmail.com> To: rack-devel@googlegroups.com References: <8F93F290D08845C89C2A7C1519A9138B@gmail.com> Message-Id: <59F7C873-D385-4F24-9EBC-37E219C452C0@gmail.com> X-Mailer: Apple Mail (2.1244.3) X-Original-Sender: neil.matatall@gmail.com X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of neil.matatall@gmail.com designates 74.125.83.49 as permitted sender) smtp.mail=neil.matatall@gmail.com; dkim=pass (test mode) header.i=@gmail.com Reply-To: rack-devel@googlegroups.com Precedence: list Mailing-list: list rack-devel@googlegroups.com; contact rack-devel+owners@googlegroups.com List-ID: X-Google-Group-Id: 486215384060 List-Post: , List-Help: , List-Archive: Sender: rack-devel@googlegroups.com List-Subscribe: , List-Unsubscribe: , Content-Type: multipart/alternative; boundary="Apple-Mail=_BC0CEA39-E8FA-4162-AE63-436A524AA6F0" --Apple-Mail=_BC0CEA39-E8FA-4162-AE63-436A524AA6F0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Josh, Thank you, I will investigate and report back. Neil On Jul 25, 2011, at 1:23 PM, Joshua Ballanco wrote: > Have you considered the possibility that memcache might be recycling = keys on you? >=20 > We had an issue a while back where we were using memcache for both = fragment caching and session storage. Occasionally, we would get = exceptions in retrieving a session, and looking at the error message it = was clear that what was retrieved from memcache was a fragment and not a = session. Unfortunately, we moved back to a cookie-based session store = before we had a chance to look deeper into the issue. What I can say is = that based on the key generation scheme we were using for sessions and = cache keys, there was effectively 0 chance that we were duplicating = keys. Instead, it seemed like memcache was re-using slots for different = keys when we started exhausting free slots. >=20 > Hope that helps! >=20 > - Joshua Ballanco > On Monday, July 25, 2011 at 2:22 PM, Neil wrote: >=20 >> While it's entirely possible that this issue is caused by some other >> factor, but we are getting session collisions as well as an issue >> where one user is getting another user's session. This is clearly >> bad, but I cannot for the life of me figure out how this could even >> happen in the first place. The code looks thread safe to me, and a >> quick discussion on #ruby-lang seems to support that. >>=20 >> Thoughts: >> 1. Session IDs are being generated in the same sequence (uses >> securerandom -> openssl which does not have a static seed) >> 2. Threads. Looks good to me. >> 3. Maybe memcached is returning something other than "STORED/ >> NOT_STORED" for @pool.add(sid, session), but the operation still >> succeeded? >> 4. Gnomes. >>=20 >> Any input is GREATLY appreciated. Please don't say "it's an RC, what >> do you expect?" :) >>=20 >>=20 >> =46rom = https://github.com/rack/rack/blob/master/lib/rack/session/memcache.rb >> def generate_sid >> loop do >> sid =3D super >> break sid unless @pool.get(sid, true) >> end >> end >>=20 >> def get_session(env, sid) >> with_lock(env, [nil, {}]) do >> unless sid and session =3D @pool.get(sid) >> sid, session =3D generate_sid, {} >> unless /^STORED/ =3D~ @pool.add(sid, session) >> raise "Session collision on '#{sid.inspect}'" >> end >> end >> [sid, session] >> end >> end >>=20 >> def set_session(env, session_id, new_session, options) >> expiry =3D options[:expire_after] >> expiry =3D expiry.nil? ? 0 : expiry + 1 >>=20 >> with_lock(env, false) do >> @pool.set session_id, new_session, expiry >> session_id >> end >> end >=20 --Apple-Mail=_BC0CEA39-E8FA-4162-AE63-436A524AA6F0 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
Have you considered the possibility that memcache = might be recycling keys on you?

We had an issue a = while back where we were using memcache for both fragment caching and = session storage. Occasionally, we would get exceptions in retrieving a = session, and looking at the error message it was clear that what was = retrieved from memcache was a fragment and not a session. Unfortunately, = we moved back to a cookie-based session store before we had a chance to = look deeper into the issue. What I can say is that based on the key = generation scheme we were using for sessions and cache keys, there was = effectively 0 chance that we were duplicating keys. Instead, it seemed = like memcache was re-using slots for different keys when we started = exhausting free slots.

Hope that = helps!

- Joshua Ballanco

On Monday, July = 25, 2011 at 2:22 PM, Neil wrote:

While it's entirely possible that = this issue is caused by some other
factor, but we are getting session = collisions as well as an issue
where one user is getting another = user's session. This is clearly
bad, but I cannot for the life of me = figure out how this could even
happen in the first place. The code = looks thread safe to me, and a
quick discussion on #ruby-lang seems = to support that.

Thoughts:
1. Session IDs are being generated = in the same sequence (uses
securerandom -> openssl which does not = have a static seed)
2. Threads. Looks good to me.
3. Maybe = memcached is returning something other than "STORED/
NOT_STORED" for = @pool.add(sid, session), but the operation still
succeeded?
4. = Gnomes.

Any input is GREATLY appreciated. Please don't say "it's = an RC, what
do you expect?" :)


=46rom https://github.com/rack/rack/blob/master/lib/rack/session/memcache.rb=
def generate_sid
loop do
sid =3D = super
break sid unless @pool.get(sid, true)
= end
end

def get_session(env, sid)
= with_lock(env, [nil, {}]) do
unless sid and session =3D = @pool.get(sid)
sid, session =3D generate_sid, {}
= unless /^STORED/ =3D~ @pool.add(sid, session)
= raise "Session collision on '#{sid.inspect}'"
end
= end
[sid, session]
end
end

= def set_session(env, session_id, new_session, options)
= expiry =3D options[:expire_after]
expiry =3D expiry.nil? ? 0 = : expiry + 1

with_lock(env, false) do
= @pool.set session_id, new_session, expiry
session_id
= end
end
=20 =20 =20 =20
=20


= --Apple-Mail=_BC0CEA39-E8FA-4162-AE63-436A524AA6F0--