From mboxrd@z Thu Jan 1 00:00:00 1970 Delivered-To: chneukirchen@gmail.com Received: by 10.229.109.203 with SMTP id k11cs72213qcp; Wed, 3 Aug 2011 03:33:20 -0700 (PDT) Return-Path: Received-SPF: pass (google.com: domain of rack-devel+bncCOCkpKnzHBDsx-TxBBoEdozTkQ@googlegroups.com designates 10.223.89.75 as permitted sender) client-ip=10.223.89.75; Authentication-Results: mr.google.com; spf=pass (google.com: domain of rack-devel+bncCOCkpKnzHBDsx-TxBBoEdozTkQ@googlegroups.com designates 10.223.89.75 as permitted sender) smtp.mail=rack-devel+bncCOCkpKnzHBDsx-TxBBoEdozTkQ@googlegroups.com; dkim=pass header.i=rack-devel+bncCOCkpKnzHBDsx-TxBBoEdozTkQ@googlegroups.com Received: from mr.google.com ([10.223.89.75]) by 10.223.89.75 with SMTP id d11mr296165fam.4.1312367599228 (num_hops = 1); Wed, 03 Aug 2011 03:33:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=beta; h=x-beenthere:received-spf:date:from:to:message-id:in-reply-to :references:subject:x-mailer:mime-version:x-original-sender :x-original-authentication-results:reply-to:precedence:mailing-list :list-id:x-google-group-id:list-post:list-help:list-archive:sender :list-subscribe:list-unsubscribe:content-type; bh=3fry9p18BCt9mg7FAIXkjDmqm8u/dnWl0ndoLBtZWZs=; b=RXqju9PWjh6dMQ1JiQPj0oTpXg5KVv7kwWPFoisRI8HfT731+NcL0TngMCl4apTgx0 D0yfanLEumMjR8mWfZT1ikHnJKr5PIY+7+M8Wr2/6FKpe1kFvAuhpAlbsxLNUGvSBknv H96p+yK78+tNwQjLI1MRWr1Z3um8Z7pdpxmGE= Received: by 10.223.89.75 with SMTP id d11mr90659fam.4.1312367596958; Wed, 03 Aug 2011 03:33:16 -0700 (PDT) X-BeenThere: rack-devel@googlegroups.com Received: by 10.204.154.210 with SMTP id p18ls286346bkw.3.gmail; Wed, 03 Aug 2011 03:33:15 -0700 (PDT) Received: by 10.204.135.22 with SMTP id l22mr851135bkt.5.1312367595540; Wed, 03 Aug 2011 03:33:15 -0700 (PDT) Received: by 10.204.135.22 with SMTP id l22mr851134bkt.5.1312367595517; Wed, 03 Aug 2011 03:33:15 -0700 (PDT) Received: from mail-fx0-f47.google.com (mail-fx0-f47.google.com [209.85.161.47]) by gmr-mx.google.com with ESMTPS id m24si327383fag.2.2011.08.03.03.33.15 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 03 Aug 2011 03:33:15 -0700 (PDT) Received-SPF: pass (google.com: domain of jballanc@gmail.com designates 209.85.161.47 as permitted sender) client-ip=209.85.161.47; Received: by fxg11 with SMTP id 11so972482fxg.20 for ; Wed, 03 Aug 2011 03:33:15 -0700 (PDT) Received: by 10.223.16.140 with SMTP id o12mr5366261faa.89.1312367583552; Wed, 03 Aug 2011 03:33:03 -0700 (PDT) Received: from DeepThought.local ([178.245.255.215]) by mx.google.com with ESMTPS id q5sm442752fah.6.2011.08.03.03.32.59 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 03 Aug 2011 03:33:02 -0700 (PDT) Date: Wed, 3 Aug 2011 13:32:57 +0300 From: Joshua Ballanco To: rack-devel@googlegroups.com Message-ID: <2CD439944A9549FA982EC7D0A8A85EA0@gmail.com> In-Reply-To: <4E31BE74.8060300@gmail.com> References: <8F93F290D08845C89C2A7C1519A9138B@gmail.com> <4E31BE74.8060300@gmail.com> Subject: Re: Session collisions on rails 3.1rc4 (authlogic, omniauth, memcache store, passenger) X-Mailer: sparrow 1.3.1 (build 814.15) MIME-Version: 1.0 X-Original-Sender: jballanc@gmail.com X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jballanc@gmail.com designates 209.85.161.47 as permitted sender) smtp.mail=jballanc@gmail.com; dkim=pass (test mode) header.i=@gmail.com Reply-To: rack-devel@googlegroups.com Precedence: list Mailing-list: list rack-devel@googlegroups.com; contact rack-devel+owners@googlegroups.com List-ID: X-Google-Group-Id: 486215384060 List-Post: , List-Help: , List-Archive: Sender: rack-devel@googlegroups.com List-Subscribe: , List-Unsubscribe: , Content-Type: multipart/alternative; boundary="4e3923d9_4db127f8_1381c" --4e3923d9_4db127f8_1381c Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Content-Disposition: inline Hi Neil, Good to hear you figured that out. I, too, had heard mention of Dalli as a replacement for memcache-client. Perhaps we should update Rack's gemspec? - Josh On Thursday, July 28, 2011 at 10:54 PM, Neil Matatall wrote: > Joshua, > > I think what you described was indeed our issue. The call to @pool.add > actually returned "END" which is a sign of a get_multikey > > We were using memcache-client and the author informed me that the > project was no longer supported and said that this can occur when the > same memcache instance handles sessions and data. He recommended > splitting the caches as well as switching to Dalli instead. > > So...not a rack issue! > > Thanks! > Neil > > On 7/25/11 1:23 PM, Joshua Ballanco wrote: > > Have you considered the possibility that memcache might be recycling > > keys on you? > > > > We had an issue a while back where we were using memcache for both > > fragment caching and session storage. Occasionally, we would get > > exceptions in retrieving a session, and looking at the error message > > it was clear that what was retrieved from memcache was a fragment and > > not a session. Unfortunately, we moved back to a cookie-based session > > store before we had a chance to look deeper into the issue. What I can > > say is that based on the key generation scheme we were using for > > sessions and cache keys, there was effectively 0 chance that we were > > duplicating keys. Instead, it seemed like memcache was re-using slots > > for different keys when we started exhausting free slots. > > > > Hope that helps! > > > > - Joshua Ballanco > > > > On Monday, July 25, 2011 at 2:22 PM, Neil wrote: > > > > > While it's entirely possible that this issue is caused by some other > > > factor, but we are getting session collisions as well as an issue > > > where one user is getting another user's session. This is clearly > > > bad, but I cannot for the life of me figure out how this could even > > > happen in the first place. The code looks thread safe to me, and a > > > quick discussion on #ruby-lang seems to support that. > > > > > > Thoughts: > > > 1. Session IDs are being generated in the same sequence (uses > > > securerandom -> openssl which does not have a static seed) > > > 2. Threads. Looks good to me. > > > 3. Maybe memcached is returning something other than "STORED/ > > > NOT_STORED" for @pool.add(sid, session), but the operation still > > > succeeded? > > > 4. Gnomes. > > > > > > Any input is GREATLY appreciated. Please don't say "it's an RC, what > > > do you expect?" :) > > > > > > > > > From > > > https://github.com/rack/rack/blob/master/lib/rack/session/memcache.rb > > > def generate_sid > > > loop do > > > sid = super > > > break sid unless @pool.get(sid, true) > > > end > > > end > > > > > > def get_session(env, sid) > > > with_lock(env, [nil, {}]) do > > > unless sid and session = @pool.get(sid) > > > sid, session = generate_sid, {} > > > unless /^STORED/ =~ @pool.add(sid, session) > > > raise "Session collision on '#{sid.inspect}'" > > > end > > > end > > > [sid, session] > > > end > > > end > > > > > > def set_session(env, session_id, new_session, options) > > > expiry = options[:expire_after] > > > expiry = expiry.nil? ? 0 : expiry + 1 > > > > > > with_lock(env, false) do > > > @pool.set session_id, new_session, expiry > > > session_id > > > end > > > end --4e3923d9_4db127f8_1381c Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
Hi Neil,

Good to hear you figured= that out. I, too, had heard mention of Dalli as a replacement for memcac= he-client. Perhaps we should update Rack's gemspec=3F

- Josh
=20

On Thursday, July 28, = 2011 at 10:54 PM, Neil Matatall wrote:

Joshua,

I think what you descr= ibed was indeed our issue. The call to =40pool.add
actually returned = =22END=22 which is a sign of a get=5Fmultikey

We were using memcac= he-client and the author informed me that the
project was no longer su= pported and said that this can occur when the
same memcache instance h= andles sessions and data. He recommended
splitting the caches as well= as switching to Dalli instead.

So...not a rack issue=21

Th= anks=21
Neil

On 7/25/11 1:23 PM, Joshua Ballanco wrote:
Have you considered the possibility that m= emcache might be recycling
keys on you=3F

We had an issue a whi= le back where we were using memcache for both
fragment caching and ses= sion storage. Occasionally, we would get
exceptions in retrieving a se= ssion, and looking at the error message
it was clear that what was ret= rieved from memcache was a fragment and
not a session. Unfortunately, = we moved back to a cookie-based session
store before we had a chance t= o look deeper into the issue. What I can
say is that based on the key = generation scheme we were using for
sessions and cache keys, there was= effectively 0 chance that we were
duplicating keys. Instead, it seeme= d like memcache was re-using slots
for different keys when we started = exhausting free slots.

Hope that helps=21

- Joshua Ballanco=

On Monday, July 25, 2011 at 2:22 PM, Neil wrote:

While it's entirely possible that this issue is= caused by some other
factor, but we are getting session collisions as= well as an issue
where one user is getting another user's session. Th= is is clearly
bad, but I cannot for the life of me figure out how this= could even
happen in the first place. The code looks thread safe to m= e, and a
quick discussion on =23ruby-lang seems to support that.
Thoughts:
1. Session IDs are being generated in the same sequence (u= ses
securerandom -> openssl which does not have a static seed)
2= . Threads. Looks good to me.
3. Maybe memcached is returning something= other than =22STORED/
NOT=5FSTORED=22 for =40pool.add(sid, session), = but the operation still
succeeded=3F
4. Gnomes.

Any input is= GREATLY appreciated. Please don't say =22it's an RC, what
do you expe= ct=3F=22 :)


=46rom
https://github.com/rack/rac= k/blob/master/lib/rack/session/memcache.rb
def generate=5Fsid
l= oop do
sid =3D super
break sid unless =40pool.get(sid, true)
end=
end

def get=5Fsession(env, sid)
with=5Flock(env, =5Bnil, =7B= =7D=5D) do
unless sid and session =3D =40pool.get(sid)
sid, session= =3D generate=5Fsid, =7B=7D
unless /=5ESTORED/ =3D=7E =40pool.add(sid,= session)
raise =22Session collision on '=23=7Bsid.inspect=7D'=22
e= nd
end
=5Bsid, session=5D
end
end

def set=5Fsession(en= v, session=5Fid, new=5Fsession, options)
expiry =3D options=5B:expire=5F= after=5D
expiry =3D expiry.nil=3F =3F 0 : expiry + 1

with=5Floc= k(env, false) do
=40pool.set session=5Fid, new=5Fsession, expiry
se= ssion=5Fid
end
end
<= /div> =20 =20 =20 =20 =20

--4e3923d9_4db127f8_1381c--