Loading...

Loading...

Corrupted queries in ColdFusion MX 7
I hate to think that there is possibly a flaw in ColdFusion...it pains me to say it. But I think there might just be a problem.
One of our latest projects was an object oriented application (lots of objects stored in memory) that sees a pretty decent load. For example, one day when we noticed a problem we had in the neighborhood of 20k connections.
After getting everything as smoothed out as possible, we are still seeing some pretty strange exceptions coming through. And for the most part they seem to deal with invalid SQL syntax of some sort of another.
Our system has quite a bit of dynamic querying going on, but I would say probably significantly less actual queries to the DB server on an average request.
It seems, that around a heavy traffic period we begin to see these SQL errors coming through. So, we feel that it could very possibly be related to memory issues...but, being pretty shorthanded we have not been able to really dig as deep into the issue as we would like.
Some of the more common exceptions:
Message: Error Executing Database Query
Detail: ... Syntax error at token 0, line 0, offset 0.
OR
Message: Error Executing Database Query
Detail: ... Invalid parameter binding(s).
And a handful of actual corrupted SQL statements that make its way through.
Something like when querying a Users table... it might say
Message: Error Executing Database Query
Detail: ... Table 'Use' not found.
...basically truncating (maybe?) the SQL statement itself.
Initially the code was suspect, but we cannot find the source of the problem and so we hit Google for some commiseration.
Notice the last few comments on this blog post
And this forum thread on Adobe.com
Is anybody else experiencing this on their applications? The rumor is this issue is not present in Scorpio, but it would be nice if we could get a fix for MX7 if it truly is a CF bug.
***EDIT*** Link to the HotFix : http://kb.adobe.com/selfservice/viewContent.do?externalId=kb402002&sliceId=1
tags:
ColdFusion
Unfortunately, I have no solution to this problem. The kind of system you describe is very similar to the one I was working on and the errors did seem to occur under heavy load.
"Out is not a NeoBodyContent!! Inconceivable!"
This is not good my friend.
That one is similar in that it happens in CF code that is perfectly valid and in that I don't have a clue to what to do against it. :(
So far I can only hope someone at Adobe has a bright moment and some spare time to fix some things...
An examination of the error data shows that the SQL gets mungled somehow from what we have written in cfquery.
However, there are reports that there is a 5-8x increase in component creation. Considering your application has a lot of objects, you probably want to move to Scorpio right away.
We are running several high traffic sites, one getting 2m page views a day. We use coldSpring and cfc object caching and tons of other memory caching. The only time I have seen the "binding" error when the server was low on memory due to some abnormal traffic. Even then I only saw it once. We run multiple instances with 1g of memory max heap for each and we cycle each instance every night. We are also running the newest CF hotfixes and Jrun 4 updater 5. I belive the drivers were updated in one of those.
Just curious: has anyone tried installing a "fresh" version of CF 7.01 (not 7.02), and then running the same apps on it to see if the problem goes away?
The reason I ask is that I've been part of teams that had apps with tens of thousands of hits per day for over a year on 7.01 which used CFC in the application scope (some in other scopes), where every param had a cfqueryparam on it.... and we never got those errors. (currently I'm working on CF 6.1 apps with similar architecture, and we don't get those errors).
I'm very curious to know if this helps anyone...
I'd want to rule out drivers first... because a) they cause 99% of db problems and b) easy to resolve.
Worth a shot I guess, but I have read similar things to what Aaron mentioned.
I've been pretty active on the adobe thread you've mentioned, and I've posted elsewhere, including on comments in Ben Forta's blog (your link above) and on the DataDirect forums. We've been trying to resolve these issues for about 6 months now.
I'll try to answer some of the questions posed above, but first I'll say what issues we are faced with at the moment...
All the issues started as soon as we started testing a refactored codebase on production. The new codebase uses application.cfc and generally made more use of cfc's (including those persisted in memory - session, server and application).
We use CFMX 7.02 on Windows 2003 Server with SQL Server 2005 (on separate boxes).
Issues:
1. Invalid parameter bindings.
First we suspected drivers - so tried all the datadirect jdbc drivers, and the Microsoft 2005 JDBC driver. This made no difference. We tried a few other things, like changing cfsql_types thinking we've got some datatype mismatch going on but no joy. Thinking it was a memory related thing we've played about with the JVM heap sizes to see if that alleviated the problem. We then decided to remove the cfqueryparam, thinking that it would eliminate the errors, albeit causing the performance to take a hit. Well, the invalid parameter binding errors didn't happen, but the SQL statement started to come through truncated - as various other people have noticed. Doh. The best solution we've found is that calling stored procs instead of using inline sql statements gets around both the invalid parameter bindings and the sql truncation issue. That's where we are with that one at the moment.
2. Illegal State Exception
This happens with code that calls custom tags or the cfsavecontent tag (which is just a custom tag) - and as somebody mentioned above these call the neobodycontent class. We've tried hard to work around this bug (we are certain both of these are bugs) by replacing custom tag calls with cfc functions. This works in the most part - but there are some uses of cfsavecontent that are just hard to work around.
Other things of note
- We've tried CFMX 7, CFMX 7.01 and CFMX 7.02
- Adobe did not manage to help us because we could not reproduce the error in an isolated test case.
- I've heard rumours also that CF8 doesn't suffer from these bugs
- I'm presently trying BlueDragon 7 to see if our code works on this server (scary)
Fun fun fun...
I've been pretty active on the adobe thread you've mentioned, and I've posted elsewhere, including on comments in Ben Forta's blog (your link above) and on the DataDirect forums. We've been trying to resolve these issues for about 6 months now...
Is it possible that that is what is going on? I am not sure why somethingl like this would happen in high traffic times?
of course, it might be like playing the numbers in dating... you ask out every girl that you see in hopes that a percentage of them, no matter how small, will say "Yes". this has nothing to do with your technique, but is merely a reflection of chance.
Now, to take that to your sitution, high traffic might not affect the CF code in any way at all. It might just be upping the chance or some error occuring because you are "playing the numbers".
I can't speak to Aaron's situation, but I can tell you that isn't possible in mine. I have gotten the error on SQL statements that, while somewhat long, are not very dynamic (no cfifs, for example).
Christopher,
Good suggestion. I will have to look into that. How much did it slow it down?
Was wondering what could stop:
UPDATE Users
SET Username="patsy"
WHERE id=433
from becoming: UPDATE Users SET Username="patsy"
on the SQL end?
Can I sue Macrobe?
Anyone seeking a recently unemployed CF developer?
Dumping the value of the getters before the query was run confirmed that each getter was retuning the correct value.
We have CFMX 7.0.2 multi-instance installed in all of our environments. We also recently updated everything to JVM 1.4.2_13 with the exception of our production servers. They still run version 1.4.2_12.
We downgraded the QA server to that version and the problem went away.
Has anyone checked their JVM?
Thanks for your feedback.
Once we finally got the load balancing configured correctly, we have seen a drastic reduction in the number of these exceptions. This obviously would have effect the application(s) in multiple ways.
Basically we now have half the object instantiations, sessions, etc that we were seeing. Which leads to less memory consumption under the same load. Could this error start occurring under heavy(er) memory use? Does it have to do with objects down in the Java level? I wonder if by increasing maz heap size, tweaking the garbage collection settings, etc (jvm tuning) this "bug" could be fixed for those of us seeing the same exceptions.
So your observation wrt sessions seems to match out intial findings. We see these exceptions coming up after and during session timeouts. We have not observed any corelation with high memory.
One of senior engineers, Ram noted following about this issue:
This is caused by following combination ...
1. application.cfc
2. App event methods in application.cfc like onSessionEnd etc.
3. Calling a method on a CFC from app event method that is stored in the app scope
4. CFML request executing at the same time when app event method is being called. And the cfml page has a custom tag
However, this has been a very difficult issue to reproduce. I will post a comment here if the fix becomes available so that you can verify and let us know if it works for you.
The HF are posted on the adobe site and can be searched via http://www.adobe.com/cfusion/knowledgebase/index.c...
Thanks,
Hemant
The combination you list is exactly consistent with our application - newly re-written with application.cfc, plenty of app persisted cfc's, cfm pages with cfmodule/custom tag calls all happening in the same request. We haven't been able to reproduce it, other than putting the codebase into production.
In reply to another comment about the JVM, we've tried many flavours of JVM and all suffer the same problem. Including 1.4.2_12.
It seems that in the last couple of weeks they have now started making real progress. So I'd keep an eye out for a patch coming soon to adobe.com.
- Mark
Thanks,
Aaron
Thanks
Ed
Thanks Ken for the hotfix, I applied it to a production server last Friday and haven't seen an illegalStateException or Invalid Parameter Binding(s) error since then.
We've been bashing away at this for the best part of a year - coming across many folks that are having the same issue. I notice that the hotfix hasn't been officially released on the Adobe website - but I'm happy to keep this one in production since the servers are significantly more stable now.
Thanks
Ed
It is worth noting here that during this time I had the opportunity to evaluate both BlueDragon 7 and CFMX8 (8 only in the last week or so). MX8 is still too buggy in beta 2 to do any good comparisons, but I was impressed with BD 7 and the new features, and also in how much support I got from New Atlanta. Ok so they clearly want us to migrate from CFMX to BD, but if it is an indication of the level of support you can expect from them then it is a ton more than you get from Adobe. I'll definately be evaluating BD7.NET when we upgrade from CFMX7...
Ken Smith
CF/JRun Server Escalation Team
Adobe Systems
Btw, doesn't 'hotfix' mean you don't have to restart anything? Because I did got the errors after applying the hotfix at first. Only after the reboot the errors went away. (I wanted to just restart CF, but a well timed 2 hour power failure in our area did effectively reboot everything :) )
That is surprising to hear. Early alphas of the release have been powering parts of Adobe.com without error (CFMX7 code) and the beta testers are saying the same.
How do you compare BD7 vs CF8? Doesn't really seem fair to New Atlanta. You should probably compare BD7 to CFMX7 (the one that came out 2.5 years ago). Functionally those version are somewhat the same (although migrating to BD usually requires a significant re-write of the application).
Either way, kudos to Adobe support for releasing a hot fix and responding the community.
- comparing MX8 and BD7. You are right, comparing the two doesn't seem fair to New Atlanta who are usually playing catch up. However, BD7 were first with CFTHREAD. From a customers point of view (me), if I upgrate from CFMX7 it will either be to MX8 or to BD7. So I will have to evaluate the two side-by-side. You could argue that it is a sideways move to BD7 from CFMX7, but I'd be spending money that could otherwise be spent on MX8 - so it's an upgrade in my eyes. We'd only do MX7 to BD7 if it offered more - which from what I've seen it probably does...
I'm emailing you offline. I'd like to hear more about the bug ur seeing in CF8 and see what we can do to fix it.
Thanks
"Where do we get info on new technotes/hotfixes these days? It's not listed here http://www.adobe.com/go/tn_17883 and I can't find a "recent technote" list anywhere in the CF support site either."
Unfortunately, it still bugs me that google searches turned up the dead link, and that the CF hot fix page has no mention of it.
Actually, not only CFTHREAD, but nearly all of the "new" features in CF8 are already in BD 7.0 or earlier releases, some as long as 5 years ago. See this blog entry for more info:
http://blog.newatlanta.com/index.cfm?mode=entry&en...
Also, note that we offer discounted upgrades from CF to BD; contact me privately if you need more details.
Sadly, that list isn't even a small fraction of the new features in CF8. I wonder if that secret 'discounted upgrade' helps to offset the cost of redeveloping a CF application to run on the knock-off platform.
But I digress.
Don't settle for cheap imitations.
Viva ColdFusion 8!
http://www.adobe.com/products/coldfusion/
One other thing I should mention for Ed--and other UK customers--watch for the launch of the New Atlanta UK web site in the next few weeks. I think you'll be pleasantly surprised at our UK pricing. Also note that we'll soon begin offering pre- and post-sales technical support from our London office.
Cheers.
We've been using CF since at least version 4.5. We upgrade a short while after a new version comes out - we're on 7.02 right now. There are a dozen or so servers running it. In short, we've a lot of experience with CF. When CFMX became unstable for us we had to appraise BD. I was pleasantly surprised, and found their platform to be a decent alternative. As Vince said, it wasn't that hard to port to BD - and we have over a million lines of CF code.
There is definately a market for CFMX competitors, if these guys didn't exist then there would be far less pressure on Adobe to bring out the features in MX that all the developers want. I'm all for a healthy market with competitors pushing each other to produce top server applications.




Loading....