mysql-proxy crash in network-mysqld.c

By 20010113 I meant 20110113 of course.

Hi, could you run the proxy in gdb, so that we can get more
information about the crash?

Run the proxy like this:

$ MYSQL_PROXY_WRAPPER="gdb --args" ./bin/mysql-proxy

and once on the gdb prompt, type

>run

once it crashes, type on the gdb promt

> thread apply all bt full

and paste here what you see.

Thanks

  Diego

On Tue, Feb 8, 2011 at 6:53 AM, Todd S
<email address hidden> wrote:
> Question #144570 on MySQL Proxy changed:
> https://answers.launchpad.net/mysql-proxy/+question/144570
>
> Todd S gave more information on the question:
> By 20010113 I meant 20110113 of course.
>
> --
> You received this question notification because you are a member of
> MySQL Proxy Discuss, which is an answer contact for MySQL Proxy.
>

--
Diego Medina
Web Developer
http://www.fmpwizard.com

Hi, and thanks for the advice. I'm a little concerned as to what the performance penalty might be running under gdb, as this is a production server on a fairly high traffic site.

Thanks!

Todd

Hi,

I understand your concerns, but it would be close to impossible to
determine the reason for the crash without some debugging information.

You said the proxy crashes even under light load, if there a way you
could send us some sample queries along with the modified rw-splitting
file? That may help us reproduce the problem.

Thanks

 Diego

On Wed, Feb 9, 2011 at 7:09 AM, Todd S
<email address hidden> wrote:
> Question #144570 on MySQL Proxy changed:
> https://answers.launchpad.net/mysql-proxy/+question/144570
>
>    Status: Answered => Open
>
> Todd S is still having a problem:
> Hi, and thanks for the advice. I'm a little concerned as to what the
> performance penalty might be running under gdb, as this is a production
> server on a fairly high traffic site.
>
> Thanks!
>
> Todd
>
> --
> You received this question notification because you are a member of
> MySQL Proxy Discuss, which is an answer contact for MySQL Proxy.
>

--
Diego Medina
Web Developer
http://www.fmpwizard.com

Diego,

Thanks for your prompt response! I am more than happy to run mysql-proxy in
gdb, but as I wrote earlier, I need to know what kind of performance penalty
we might be looking at. If it's "negligible" to <10% it is probably
acceptable. This is something I can't launch without some advance knowledge
because - as I wrote earlier - it is on our production server.

Best,
  Todd Steinwart
   DBA/Sysadmin Statsheet Inc.

On Wed, Feb 9, 2011 at 7:51 AM, fmpwizard <
<email address hidden>> wrote:

> Your question #144570 on MySQL Proxy changed:
> https://answers.launchpad.net/mysql-proxy/+question/144570
>
> Status: Open => Answered
>
> fmpwizard proposed the following answer:
> Hi,
>
> I understand your concerns, but it would be close to impossible to
> determine the reason for the crash without some debugging information.
>
> You said the proxy crashes even under light load, if there a way you
> could send us some sample queries along with the modified rw-splitting
> file? That may help us reproduce the problem.
>
> Thanks
>
> Diego
>
> On Wed, Feb 9, 2011 at 7:09 AM, Todd S
> <email address hidden> wrote:
> > Question #144570 on MySQL Proxy changed:
> > https://answers.launchpad.net/mysql-proxy/+question/144570
> >
> > Status: Answered => Open
> >
> > Todd S is still having a problem:
> > Hi, and thanks for the advice. I'm a little concerned as to what the
> > performance penalty might be running under gdb, as this is a production
> > server on a fairly high traffic site.
> >
> > Thanks!
> >
> > Todd
> >
> > --
> > You received this question notification because you are a member of
> > MySQL Proxy Discuss, which is an answer contact for MySQL Proxy.
> >
>
>
> --
> Diego Medina
> Web Developer
> http://www.fmpwizard.com
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
>
> https://answers.launchpad.net/mysql-proxy/+question/144570/+confirm?answer_id=3
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
> https://answers.launchpad.net/mysql-proxy/+question/144570
>
> You received this question notification because you are a direct
> subscriber of the question.
>

Hi Todd,

I just checked with some of our developers and they all agree that the
overhead will be almost none. They reminded me how many
developers/dbas use gdb as a poor's man profiler.

Thanks

  Diego

On Wed, Feb 9, 2011 at 11:15 AM, Todd S
<email address hidden> wrote:
> Question #144570 on MySQL Proxy changed:
> https://answers.launchpad.net/mysql-proxy/+question/144570
>
>    Status: Answered => Open
>
> Todd S is still having a problem:
> Diego,
>
> Thanks for your prompt response! I am more than happy to run mysql-proxy in
> gdb, but as I wrote earlier, I need to know what kind of performance penalty
> we might be looking at. If it's "negligible" to <10% it is probably
> acceptable. This is something I can't launch without some advance knowledge
> because - as I wrote earlier - it is on our production server.
>
> Best,
>  Todd Steinwart
>   DBA/Sysadmin Statsheet Inc.
>
> On Wed, Feb 9, 2011 at 7:51 AM, fmpwizard <
> <email address hidden>> wrote:
>
>> Your question #144570 on MySQL Proxy changed:
>> https://answers.launchpad.net/mysql-proxy/+question/144570
>>
>>     Status: Open => Answered
>>
>> fmpwizard proposed the following answer:
>> Hi,
>>
>> I understand your concerns, but it would be close to impossible to
>> determine the reason for the crash without some debugging information.
>>
>> You said the proxy crashes even under light load, if there a way you
>> could send us some sample queries along with the modified rw-splitting
>> file? That may help  us reproduce the problem.
>>
>> Thanks
>>
>>  Diego
>>
>> On Wed, Feb 9, 2011 at 7:09 AM, Todd S
>> <email address hidden> wrote:
>> > Question #144570 on MySQL Proxy changed:
>> > https://answers.launchpad.net/mysql-proxy/+question/144570
>> >
>> >    Status: Answered => Open
>> >
>> > Todd S is still having a problem:
>> > Hi, and thanks for the advice. I'm a little concerned as to what the
>> > performance penalty might be running under gdb, as this is a production
>> > server on a fairly high traffic site.
>> >
>> > Thanks!
>> >
>> > Todd
>> >
>> > --
>> > You received this question notification because you are a member of
>> > MySQL Proxy Discuss, which is an answer contact for MySQL Proxy.
>> >
>>
>>
>> --
>> Diego Medina
>> Web Developer
>> http://www.fmpwizard.com
>>
>> --
>> If this answers your question, please go to the following page to let us
>> know that it is solved:
>>
>> https://answers.launchpad.net/mysql-proxy/+question/144570/+confirm?answer_id=3
>>
>> If you still need help, you can reply to this email or go to the
>> following page to enter your feedback:
>> https://answers.launchpad.net/mysql-proxy/+question/144570
>>
>> You received this question notification because you are a direct
>> subscriber of the question.
>>
>
> --
> You received this question notification because you are a member of
> MySQL Proxy Discuss, which is an answer contact for MySQL Proxy.
>

--
Diego Medina
Web Developer
http://www.fmpwizard.com

Diego,

I apologize for continuing to hammer you with questions.

Would we have to run the "thread apply all bt full" command before the
server restarts? (Taking "keepalive" out of the config file is not an
option. If the site goes down during the middle of the night we'd probably
not know until 5am or so.) The 3 minute wait doesn't appear to be
configurable, although I might be able to compile with a higher value to
give us a bigger window.

Would the "log-backtrace-on-crash" option provide any (additional) useful
information?

Thanks!
  Todd

On Wed, Feb 9, 2011 at 11:42 AM, fmpwizard <
<email address hidden>> wrote:

> Your question #144570 on MySQL Proxy changed:
> https://answers.launchpad.net/mysql-proxy/+question/144570
>
> Status: Open => Answered
>
> fmpwizard proposed the following answer:
> Hi Todd,
>
> I just checked with some of our developers and they all agree that the
> overhead will be almost none. They reminded me how many
> developers/dbas use gdb as a poor's man profiler.
>
> Thanks
>
> Diego
>
> On Wed, Feb 9, 2011 at 11:15 AM, Todd S
> <email address hidden> wrote:
> > Question #144570 on MySQL Proxy changed:
> > https://answers.launchpad.net/mysql-proxy/+question/144570
> >
> > Status: Answered => Open
> >
> > Todd S is still having a problem:
> > Diego,
> >
> > Thanks for your prompt response! I am more than happy to run mysql-proxy
> in
> > gdb, but as I wrote earlier, I need to know what kind of performance
> penalty
> > we might be looking at. If it's "negligible" to <10% it is probably
> > acceptable. This is something I can't launch without some advance
> knowledge
> > because - as I wrote earlier - it is on our production server.
> >
> > Best,
> > Todd Steinwart
> > DBA/Sysadmin Statsheet Inc.
> >
> > On Wed, Feb 9, 2011 at 7:51 AM, fmpwizard <
> > <email address hidden>> wrote:
> >
> >> Your question #144570 on MySQL Proxy changed:
> >> https://answers.launchpad.net/mysql-proxy/+question/144570
> >>
> >> Status: Open => Answered
> >>
> >> fmpwizard proposed the following answer:
> >> Hi,
> >>
> >> I understand your concerns, but it would be close to impossible to
> >> determine the reason for the crash without some debugging information.
> >>
> >> You said the proxy crashes even under light load, if there a way you
> >> could send us some sample queries along with the modified rw-splitting
> >> file? That may help us reproduce the problem.
> >>
> >> Thanks
> >>
> >> Diego
> >>
> >> On Wed, Feb 9, 2011 at 7:09 AM, Todd S
> >> <email address hidden> wrote:
> >> > Question #144570 on MySQL Proxy changed:
> >> > https://answers.launchpad.net/mysql-proxy/+question/144570
> >> >
> >> > Status: Answered => Open
> >> >
> >> > Todd S is still having a problem:
> >> > Hi, and thanks for the advice. I'm a little concerned as to what the
> >> > performance penalty might be running under gdb, as this is a
> production
> >> > server on a fairly high traffic site.
> >> >
> >> > Thanks!
> >> >
> >> > Todd
> >> >
> >> > --
> >> > You received this question notification because you are a member of
> >> > MySQL Proxy Discuss, which is an answer contact for MySQL Proxy.
> >> >
> >>
> >>
> >> --
> >> Diego Medina
> >> Web Developer
> >> http://www.fmpwizard.com
> >>
> >> --
> >> If this answers your question, please go to the following page to let us
> >> know that it is solved:
> >>
> >>
> https://answers.launchpad.net/mysql-proxy/+question/144570/+confirm?answer_id=3
> >>
> >> If you still need help, you can reply to this email or go to the
> >> following page to enter your feedback:
> >> https://answers.launchpad.net/mysql-proxy/+question/144570
> >>
> >> You received this question notification because you are a direct
> >> subscriber of the question.
> >>
> >
> > --
> > You received this question notification because you are a member of
> > MySQL Proxy Discuss, which is an answer contact for MySQL Proxy.
> >
>
>
> --
> Diego Medina
> Web Developer
> http://www.fmpwizard.com
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
>
> https://answers.launchpad.net/mysql-proxy/+question/144570/+confirm?answer_id=5
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
> https://answers.launchpad.net/mysql-proxy/+question/144570
>
> You received this question notification because you are a direct
> subscriber of the question.
>

Todd,

No worries, I understand your current situation. We do need the thread
apply all command to be issue before the server restart, even more, I
think that if you use the keep alive=on, gdb will actually only track
the "angel" process, instead of the proxy process, so it will not
really do any good for us.

I'm leaning more to the idea that we need a way to reproduce this
crash outside your production system. So we could try a few fixes
without harming your site.

Would you be able to just run another proxy process on your server,
use the rw-splitting script, and send some control number of queries,
to see if we can get the crash on a second setup?

Thanks

  Diego

On Wed, Feb 9, 2011 at 1:30 PM, Todd S
<email address hidden> wrote:
> Question #144570 on MySQL Proxy changed:
> https://answers.launchpad.net/mysql-proxy/+question/144570
>
>    Status: Answered => Open
>
> Todd S is still having a problem:
> Diego,
>
> I apologize for continuing to hammer you with questions.
>
> Would we have to run the "thread apply all bt full" command before the
> server restarts? (Taking "keepalive" out of the config file is not an
> option. If the site goes down during the middle of the night we'd probably
> not know until 5am or so.) The 3 minute wait doesn't appear to be
> configurable, although I might be able to compile with a higher value to
> give us a bigger window.
>
> Would the "log-backtrace-on-crash" option provide any (additional) useful
> information?
>
> Thanks!
>  Todd
>
> On Wed, Feb 9, 2011 at 11:42 AM, fmpwizard <
> <email address hidden>> wrote:
>
>> Your question #144570 on MySQL Proxy changed:
>> https://answers.launchpad.net/mysql-proxy/+question/144570
>>
>>    Status: Open => Answered
>>
>> fmpwizard proposed the following answer:
>> Hi Todd,
>>
>> I just checked with some of our developers and they all agree that the
>> overhead will be almost none. They reminded me how many
>> developers/dbas use gdb as a poor's man profiler.
>>
>> Thanks
>>
>>  Diego
>>
>> On Wed, Feb 9, 2011 at 11:15 AM, Todd S
>> <email address hidden> wrote:
>> > Question #144570 on MySQL Proxy changed:
>> > https://answers.launchpad.net/mysql-proxy/+question/144570
>> >
>> >    Status: Answered => Open
>> >
>> > Todd S is still having a problem:
>> > Diego,
>> >
>> > Thanks for your prompt response! I am more than happy to run mysql-proxy
>> in
>> > gdb, but as I wrote earlier, I need to know what kind of performance
>> penalty
>> > we might be looking at. If it's "negligible" to <10% it is probably
>> > acceptable. This is something I can't launch without some advance
>> knowledge
>> > because - as I wrote earlier - it is on our production server.
>> >
>> > Best,
>> >  Todd Steinwart
>> >   DBA/Sysadmin Statsheet Inc.
>> >
>> > On Wed, Feb 9, 2011 at 7:51 AM, fmpwizard <
>> > <email address hidden>> wrote:
>> >
>> >> Your question #144570 on MySQL Proxy changed:
>> >> https://answers.launchpad.net/mysql-proxy/+question/144570
>> >>
>> >>     Status: Open => Answered
>> >>
>> >> fmpwizard proposed the following answer:
>> >> Hi,
>> >>
>> >> I understand your concerns, but it would be close to impossible to
>> >> determine the reason for the crash without some debugging information

I solved this problem by increasing the event_threads parameter to the number of front-end connections plus 1. We are, however, having a far more serious problem but I'll start a new thread for that.