SPIQ/QIS The Problem The company in question develops hardware and software. They have two software groups, each with circa 15 developers. Part of the system is developed in SDL. In order to focus their V&V work better they needed to know which SDL module characteristics that caused errors. Possible candidates were number of module states, number of input signals etc.

SPIQ/QIS What to measure? In order to get a consistent and efficient data collection and analysis, we started with a GQM process. This consisted of A half-day basic course in GQM. A half-day workshop where they identified –the questions that needed answers. –which metrics they needed in order to answer these questions.

SPIQ/QIS Analyser PMC- modulen for å redusere Vedlikeholds kostnadene Sett fra FW vedlikehold og utvikling Omgivelse Omskriving av PMC Fokus 1.Antall feil pr. SDL blokk 2.Rettekostnader 3.Fordeling av antall feil på kategorier 4.Fordeling av kostnader over faser 5.Antall ”nye” feil pr. SDL blokk 6.Fordeling av feil meldt internt vs. feil meldt fra eksterne kunder Omgivelser a.Antall ”fremmede” endringer pr. blokk b.Kompleksitet av blokkomgivelser c.Kompleksitet av blokk d.Volum av blokk Hva tror vi Q1_1: Ca. 400 feil hittil for SDL-delen av PMC i alt Q1_2 Q1_3: Se definisjonen av Q1_3. Q1_4: TBD Q1_5: Ca. 5% ”nye” feil Q1_6: Ca 90% av feilene meldt internt Påvirkning Økt Q1_a => Økt Q1_1, Q1_2 Økt Q1_b => Økt Q1_1, Q1_2, Q1_5 Økt Q1_c => Økt Q1_1, Q1_2, Q1_5 Økt Q1_d => Økt Q1_1, Q1_2, Q1_5 Feedback

SPIQ/QIS SpørsmålQ1_c: Kompleksiteten av blokk Tilknyttede metrikkerM5: Antall tilstander pr. blokk M6: Antall signaler inn pr. blokk (M6a og M6b) M7: Antall GOTO pr. blokk M8: Antall variable pr. blokk M9: Antall eksterne signaler ut M12: Kompleksiteten til en SDL-blokk Definisjon Presentasjon og analyseQ1_c presenters som et Kiviat-diagram med en akse for hver av de seks metrikkene M5 – M9. M12 vises ved siden som en kontrollverdi. Hva tror vi

SPIQ/QIS The Metrics Some of the metrics that were collected for each SDL module: Number of errors found. Subjective - perceived - complexity. Number of pages of SDL description. Number of SDL states Number of signals in Number of signals out

SPIQ/QIS

SPIQ/QIS

SPIQ/QIS What makes it complex - 1 In order to answer this question we went through all the hypothesis put forward by the developers during the GQM session. We will look at three of them: Number of states - M5 Number of signals out - M9 Number of pages in the SDL description - M10

SPIQ/QIS What makes it complex - 2 The identified problems were well suited for ANOVA analysis. The data for the three metrics M5, M9 and M10 were sorted according to the complexity scores (High, Medium and Low). An ANOVA analyses was then performed for each data set. We decided to require p-value better than 10%

SPIQ/QIS ANOVA results - 1 Number of states: Source of VariationSSdfMSP-value Between Groups1190, ,02080, Within Groups1631, ,3667 Number of states does not contribute significantly to the complexity as perceived by the developers.

SPIQ/QIS ANOVA results - 2 Number of signals out: Source of VariationSSdfMSP-value Between Groups2779, ,5210, Within Groups1813, ,7667 Number of signals out contribute significantly to the complexity as perceived by the developers

SPIQ/QIS ANOVA results - 3 Number of pages in the SDL description: Source of VariationSSdfMSP-value Between Groups5586, ,0210, Within Groups2133, ,7667 Number of pages in the SDL description contribute significantly to the complexity as perceived by the developers

SPIQ/QIS Summary - 1 SDL module complexity as perceived by the developers depends on two factors: Number of signals out Number of pages in the SDL description The other suspected factors identified during the GQM process did not give a significant contribution.

SPIQ/QIS What about Errors Now that we have some ideas on what makes a module look complex to the developers, the next step was to see if there is any connection between module complexity and the number of errors in the modules. As before, the ANOVA can give us an answer.

SPIQ/QIS Complexity and Errors - 1 Errors and complexity Source of VariationSSdfMSFP-value Between Groups1646, ,41670, Within Groups770, ,1333 It is reasonable to assume that complex modules have more errors.

SPIQ/QIS Complexity and Errors - 2 If we look at the summary table form ANOVA, we see that the differences are quite large: GroupsCountSumAverageVariance Column Column 23227, ,33333 Column However, due to few observations for each complexity level, the variances are large also. Thus, we should not be too categorical.

SPIQ/QIS Conclusions With all the necessary caveats in mind the company decided as follows: In order to reduce the number of errors we need to single out modules with : Large descriptions - more than 35 pages of SDL description. Many signals out - more than 30. The limiting values are the average values from the ANOVA summary tables.