+- MISRA Discussion Forums (https://forum.misra.org.uk)
+-- Forum: MISRA C (https://forum.misra.org.uk/forumdisplay.php?fid=4)
+--- Forum: General Questions (https://forum.misra.org.uk/forumdisplay.php?fid=27)
+--- Thread: Validation of MISRA? (/showthread.php?tid=423)
Validation of MISRA? - acm - 04-12-2007
It is clear that implementing the MISRA code guidelines costs a lot in training, and that it bloats source code to some extent. Do the guidelines provide more benefit than their cost? I am somewhat sceptical about these benefits.
The MISRA guidelines have been in use for around a decade. There will thus surely have been some studies undertaken that demonstrate that the MISRA guidelines work, for some value of \"work\". Or, more accurately put, that determine in which circumstances they are useful and in which they are not.
I am disappointed not to be able to find any references to these studies on the MISRA web site. I wasn't able to find any with an internet search (though my googling skills are not that good).
Would somebody please point out these studies to me?
Thanks in advance!
--
Alan Mackenzie (Nuremberg, Germany).
reports - MMouse - 05-12-2007
This is a good point.
The problem is most of the companies using them are not research places. They are busy.
However there are many studies that show using a subset of C, a style guide and static analysis does improve projects with requard to fewer errors and more reliable code, often in reduced timescales.
However this is only when the subset is used as part of a good process. Drinking diet coke will not help much if you are eating it with a large buger and fries.
There are many studies around on coding standards. MISRA-C is but one of many coding standards and the results of these studies will broadly apply to using MISRA-C
Please provide references - derek_farn - 05-12-2007
Can you please provide references to some of these \"many studies\".
I have looked long and hard for articles/papers that have studied the impact of coding guidelines/language subsets/tools on software quality/errors/maintenance.
I know of very little work in this area and what work has been done is sketchy at best.
I would be very interested to read about the studies you know about.
Re: Please provide references - MMouse - 17-12-2007
derek_farn Wrote:Can you please provide references to some of these "many studies".
I have looked long and hard for articles/papers that have studied the impact of coding guidelines/language subsets/tools on software quality/errors/maintenance.
I know of very little work in this area and what work has been done is sketchy at best.
I would be very interested to read about the studies you know about.
http://www.knosof.co.uk/cbook/guidelinedocs.html
Cited reference does not contain requested information - derek_farn - 17-12-2007
A little extra checking of the reference you just posted would show:
o It does not contain any references to studies that perform a cost/benefit analysis of coding guidelines.
o I am the author of the bibliography you cited (ok, I forgot to include my name on that particular page, but its address matches that of my signature).
- acm - 18-12-2007
Hello again, MMouse and Derek.
Thanks indeed for your replies to my post.
From MMouse:
> However this is only when the subset is used as part of a good process.
> Drinking diet coke will not help much if you are eating it with a large
> buger and fries.
I think this is the point. It is the process (and the quality of the engineers, of course) that produces good and safe software. I doubt that the specific rules in MISRA are, in themselves, of much importance - any such coding guidelines, or perhaps none at all, would be just as good.
Oh, by the way, I once had a diet coke in the company of a large bugler - very good company he was, when not blowing his own trumpet. ;-)
So, I challenge the promulgators and maintainers of the MISRA standard: Please show me some evidence that your standard has some actual value in software engineering, and isn't just a .
--
Alan Mackenzie (Nuremberg, Germany).
coding standards - MMouse - 19-12-2007
Gruss Gott Alan,
To answer some of your points:
Training:- New programmers need training anyway. A good coding standard stops bad habits and promotes a much better Engineering understanding of the language.
Far too many have bad habits that look \"cool!\" which store up trouble later on. So Training on MISRA is not a cost as it saves time later. I can give anecdotal evidence for this.
Actually it is not just new programmers but old ones too... Besides using an automated static analyser should pick up most of the MISRA rules anyway... So the programmers are trained by the tools and this does not cost time or effort.
Code bloat: Space in source code is free. Space in the binary is important In fact I will dig out an example I have of a piece of code that is twice as long with more variables and another more compact, less readable, piece of code that does the same thing.
Not only was the resultant binary the same size but the one with the longer source was much easier to debug both at source and assembler level using an ICE.
I would expect that within a few weeks the MISRA team will produce a paper or something to show that MISRA C does save time/money/errors etc since you have raised a the spectre of the Emperors New Clothes! They have to show that the emperor IS wearing new cloths and it is not just a con. Especially as they are now starting MISRA-C 3
On the other side what evidence do you have that MISRA-C DOES bloat code and DOES cost (overall) for training etc
Best wishes for the Season
MMouse
- derek_farn - 19-12-2007
Quote:Training:- New programmers need training anyway. A good coding standard stops bad habits and promotes a much better Engineering understanding of the language.I think this statement is self evident. Now where is the evidence that MISRA C is a good coding standard?
Quote:Training on MISRA is not a cost as it saves time later.Training is a cost, saving time later is a benefit. A cost/benefit analysis would evaluate whether the cost was worth the benefit. It might also evaluate whether the same cost could be spent on another kind of training for a greater benefit.
Quote:Besides using an automated static analyser should pick up most of the MISRA rules anyway... So the programmers are trained by the tools and this does not cost time or effort.It will take a finite amount of time to read the messages generated by a tool, understand them and act n them. This is a cost, there may be a benefit as a result of investing in this cost. Again where is the cost/benefit analysis?
Quote:Code bloat: Space in source code is free. Space in the binary is important In fact I will dig out an example I have of a piece of code that is twice as long with more variables and another more compact, less readable, piece of code that does the same thing.Ok, you have such code, but what does it have to do with MISRA C?
Quote:On the other side what evidence do you have that MISRA-C DOES bloat code and DOES cost (overall) for training etc.Ok, so you have given up on your previous statement that there have been many studies showing the benefits of MISRA C.
I make no claims that MISRA C bloats code. Overall I think MISRA C is neutral with regard to code bloat, ie not bigger and not smaller.
With regard to training costs. Let's assume it takes somebody 10 minutes to read and understand a MISRA C rule, with say 120 rules that is 20 hours of time. I don't know many people who would put a zero cost on 20 hours of time. Where is the analysis showing that using MISRA C has a benefit of at least 20 hours of somebodies time?
- acm - 19-12-2007
Hi MM!
>Training:- New programmers need training anyway. A good coding standard stops bad habits and promotes a much better Engineering understanding of the language.
Agreed. Which is why it's important to ensure that MISRA is, in fact, a _good_ standard, for some value of \"good\". I'm asking in this post for some, any, evidence of this goodness.
> Far too many have bad habits that look \"cool!\" which store up trouble later on. So Training on MISRA is not a cost as it saves time later. I can give anecdotal evidence for this.
As can I. But many (most?) of these aren't covered by MISRA; Overlong names, deeply nested structs, typedeffing lots of different sorts of integer.... I think that no matter how comprehensive you make coding standards, somebody will find a way to write grotty code whilst conforming to them. MISRA, in this respect, looks reasonable, up to a point.
> Code bloat: Space in source code is free.
I don't agree. Only so many lines will fit on a video screen at a time. A piece of code which you have to scroll up and down to understand or debug is more difficult (hence more liable to bugs) than a concise equivalent. Mentally filtering out clutter whilse reading code is hard work.
I read somewhere (it may have been in Steve McConnell's book about coding) that bug frequency / 1000 lines of code is pretty much independent of the source language. It therefore seems plausible that code written in \"MISRA-C\" will contain more bugs than the same stuff written in full C. This is the sort of issue that the putative studies should address.
[ .... ]
> I would expect that within a few weeks the MISRA team will produce a paper or something to show that MISRA C does save time/money/errors etc since you have raised a the spectre of the Emperors New Clothes! They have to show that the emperor IS wearing new cloths and it is not just a con. Especially as they are now starting MISRA-C 3
Good! That is what I want.
>On the other side what evidence do you have that MISRA-C DOES bloat code and DOES cost (overall) for training etc
I think both are self evident. Note that I used the word \"somewhat\" when I mentioned bloat. Examples are rule 14.8 (The statement part of \"if\" etc., may only be a compound statement) and rule 14.7 (Only one \"return\" in a function). So that perfectly idiomatic C like
if (foo > MAX_FOO)
return EXCESSIVE_FOO;
gets bloated to
int ret_value;
....
if (foo > MAX_FOO)
{
ret_value = EXCESSIVE_FOO;
}
else
{
} /* or some other similar coding strategem */
return ret_value;
However, I don't think it would be productive to get into an argument about detailed coding practices in this thread. But I would like some evidence that the bit of bloat caused by these rules actually brings more benefit than cost.
As for the training, I've had to learn how to use my firm's chosen tools for detecting and \"fixing\" the things it points out. That's time I would rather have spent debugging.
>Best wishes for the Season
And yourself, sir!
>MMouse
--
Alan Mackenzie (Nuremberg, Germany)
- acm - 04-01-2008
Derek Farn wrote:
>Can you please provide references to some of these \"many studies\".
>I have looked long and hard for articles/papers that have studied the impact of coding guidelines/language subsets/tools on software quality/errors/maintenance.
I've been doing some Googling over the holidays. Dr. Les Hatton (the guy who wrote \"Safer C\") has done actual research on the topic. His home page is at http://www.leshatton.org.
Interestingly, he divides coding guidelines into types A (purely style) and B (safety related). He further divides B guidelines into B.1 (\"folklore\"; e.g. the prohibition of goto) and B.2 (those for which there's evidence, no matter how slight; e.g., casting pointer types).
According to Dr. Hatton, only 55 of the 127 rules in MISRA-1 were of type B.2 (though he doesn't identify them ;-). He isn't too impressed with MISRA-2 either - he seems to regard it as an opportunity lost.
--
Alan Mackenzie (Nuremberg, Germany).
Evidence of a benefit, but not of cost/benefit - derek_farn - 04-01-2008
Alan,
I know Les, he is the head of department at Kingston University where I am a visiting professor.
Les's work showed that the some of the constructs that are recommended against by the MISRA guidelines (his work predated the MISRA and so does not address them directly) occur in in real code and are a source of faults. So there is a benefit to avoiding them.
What Les did not do is provide a cost/benefit analysis of the guidelines.
Re: - MMouse - 04-01-2008
acm Wrote:Hi MM!
> Code bloat: Space in source code is free.
I don't agree. Only so many lines will fit on a video screen at a time. A piece of code which you have to scroll up and down to understand or debug is more difficult (hence more liable to bugs) than a concise equivalent. Mentally filtering out clutter whilse reading code is hard work.
I agree BUT the same is true of compact code that is complex. Taken to either extream it is not good. However there is a tendancy to be "too clever" when trying to save space on screen. The result is less readable code and more errors.
Besides these days with 19 in LCD screens being the norm from 14 inch (real view 12") CRT screens 135 columns is more common than 40 or 80 colums. Though with graphic secreens this is no longer the measurement it once was.
acm Wrote:I read somewhere (it may have been in Steve McConnell's book about coding) that bug frequency / 1000 lines of code is pretty much independent of the source language. It therefore seems plausible that code written in "MISRA-C" will contain more bugs than the same stuff written in full C. This is the sort of issue that the putative studies should address.
Bugs per line of code is a good measure. However you have to measure like for like. Some projects have a much lower B/KLOC than others. One would expect MISRA-C compliant code to have a low B/KLOC
acm Wrote:> I would expect that within a few weeks the MISRA team will produce a paper or something to show that MISRA C does save time/money/errors etc since you have raised a the spectre of the Emperors New Clothes! They have to show that the emperor IS wearing new cloths and it is not just a con. Especially as they are now starting MISRA-C 3
Good! That is what I want.
You and me both :-)
acm Wrote:>On the other side what evidence do you have that MISRA-C DOES bloat code and DOES cost (overall) for training etcI take your point that the source code get bigger on the page. However it is also more readable and in some cases easier to debug. I will see If I can find the example I did years ago where more compact code was less easy to debug in an emulator.
I think both are self evident. Note that I used the word "somewhat" when I mentioned bloat. Examples are rule 14.8 (The statement part of "if" etc., may only be a compound statement) and rule 14.7 (Only one "return" in a function). So that perfectly idiomatic C like
if (foo > MAX_FOO)
return EXCESSIVE_FOO;
gets bloated to
int ret_value;
....
if (foo > MAX_FOO)
{
ret_value = EXCESSIVE_FOO;
}
else
{
} /* or some other similar coding strategem */
return ret_value;
However, I don't think it would be productive to get into an argument about detailed coding practices in this thread. But I would like some evidence that the bit of bloat caused by these rules actually brings more benefit than cost.
acm Wrote:As for the training, I've had to learn how to use my firm's chosen tools for detecting and "fixing" the things it points out. That's time I would rather have spent debugging.
Alan Mackenzie (Nuremberg, Germany)
Now this I must take issue with. A good static analyser will, after you have set it up and spent time being trained on it, catch a hell of a lot of bugs. The poit is it is far more cost effective to find bugs with a static analyser just after they have been writen than woorking out what just happened in a test/debug session.
There are several reports on the telecoms industry that show that good use of static analysis will reduce the bug hunting phase deastically and give more time for additional testing. So you get a better tested product in a shorter space of time because more of the problems were picked up durring coding and the bug hunting time was much shorter.
Happy new year.
Chris
- acm - 08-01-2008
Hi, MM!
>
acm Wrote:I read somewhere (it may have been in Steve McConnell's book about coding) that bug frequency / 1000 lines of code is pretty much independent of the source language. It therefore seems plausible that code written in "MISRA-C" will contain more bugs than the same stuff written in full C. This is the sort of issue that the putative studies should address.
>Bugs per line of code is a good measure. However you have to measure like for like. Some projects have a much lower B/KLOC than others. One would expect MISRA-C compliant code to have a low B/KLOC
Well would one, now? That's begging the whole point of this thread. :-) Where is the evidence that the (moderate) code-bloat mandated by MISRA is innocuous? In the example I gave, it was quite marked, and in my judgement harmful.
>Now this I must take issue with [time acm spent learning MISRA tool was not well spent]. A good static analyser will, after you have set it up and spent time being trained on it, catch a hell of a lot of bugs. The point is it is far more cost effective to find bugs with a static analyser just after they have been writen than woorking out what just happened in a test/debug session.
I agree, if we're talking about a GOOD static analyser. This needs a good basis to build on. Is MISRA such a basis? The tool I had to learn had a very low "signal/nose ratio" indeed, so much so that the time I've spent using it hasn't been fruitful. I've been using it on old code, though, not new code.
>There are several reports on the telecoms industry that show that good use of static analysis will reduce the bug hunting phase deastically ....
What's deism got to do with it? :-)
Cheers!
--
Alan Mackenzie (Nuremberg, Germany)