How can I get F statistic values for an ANOVA in R?

Question

I am currently running some algorithms to solve a multi-objective linear mathematical model (Operation Research). I've used three algorithms: Constraint Method (C-M), Non-Sorting Genetic Algorithm II (NSGA-II) and Strength Pareto Evolutionary Algorithm 2 (SPEA2). I've set a performance indicator as to the number of the solution in the Pareto Border (No_solutions) and I gathered data from six instances (I1,..,I6).

My data is the following table:

   Instance Algorithm No_solutions

 1 I1       C-M                 48
 2 I2       C-M                 46
 3 I3       C-M                 51
 4 I4       C-M                 50
 5 I5       C-M                 51
 6 I6       C-M                 49
 7 I1       NSGA-II            300
 8 I2       NSGA-II            300
 9 I3       NSGA-II            300
10 I4       NSGA-II            300
11 I5       NSGA-II            300
12 I6       NSGA-II            300
13 I1       SPEA2              150
14 I2       SPEA3              150
15 I3       SPEA4              150
16 I4       SPEA5              150
17 I5       SPEA6              150
18 I6       SPEA7              150

I've run an ANOVA in R using aov(). My code is the following:

performance_1 <- read_excel("C:/Users/Alonso/Desktop/metaheuristic_comparation/performance.xlsx")
View(performance_1)
attach(performance_1)
names(performance_1)
str(performance_1)

Factor_algorithm <- factor(Algorithm)
Factor_instance <- factor(Instance)
Respuesta <- performance_1$No_solutions

Modelo <-lm(Respuesta ~ (Factor_algorithm+Factor_instance)^2)
ANOVA <-aov(Modelo)
summary(ANOVA)

The output is:

 Modelo <-lm(Respuesta ~ (Factor_algorithm+Factor_instance)^2)
 ANOVA <-aov(Modelo)
 summary(ANOVA)
                                 Df Sum Sq Mean Sq
Factor_algorithm                  7 191169   27310
Factor_instance                   5      9       2
Factor_algorithm:Factor_instance  5      9       2

I've read the documentation of aov() method and I am pretty sure that the F-statistic must be shown. Any ideas? I am pretty sure that I can obtain more information using this method

Answer 1

Is there an error in your data? There are 8 algorithms indicated, compared to 3 in your question text. If you change all SPEA# to SPEA2 and drop the interaction you can get F-values, but it's still a non-sense model considering the data and the within-/between-group variances. Algorithm is trivially significant, while Instance is trivially not.

pp <- read.table(text="
   Instance Algorithm No_solutions
 1 I1       C-M                 48
 2 I2       C-M                 46
 3 I3       C-M                 51
 4 I4       C-M                 50
 5 I5       C-M                 51
 6 I6       C-M                 49
 7 I1       NSGA-II            300
 8 I2       NSGA-II            300
 9 I3       NSGA-II            300
10 I4       NSGA-II            300
11 I5       NSGA-II            300
12 I6       NSGA-II            300
13 I1       SPEA2              150
14 I2       SPEA2              150
15 I3       SPEA2              150
16 I4       SPEA2              150
17 I5       SPEA2              150
18 I6       SPEA2              150")

anova(aov(No_solutions ~ Instance+Algorithm, data=pp))
# Analysis of Variance Table
# 
# Response: No_solutions
#           Df Sum Sq Mean Sq F value Pr(>F)    
# Instance   5      6       1       1 0.4651    
# Algorithm  2 191169   95585   76129 <2e-16 ***
# Residuals 10     13       1                   

Answer 2

Take a closer look at the documentation for the aov function. It states that F and p-values are only shown if you have non-zero residual degrees of freedom in your model

In your case you have 18 data points, your predictors are using up 17 (7+5+5) degrees of freedom, and 1 is required for the model itself, which brings you to 18. This leaves you with 0 residual degrees of freedom and therefore the F-value is not displayed. The solution is to collect more data or simplify your model