Problems with convergency when using solnp function in R

Question

When solving a portfolio optimization problem with an additional 1-norm constraint on the portfolio weights, I faced with convergency problems.

Description of the exercise:

For given N assets with T observations of their returns find the value of 1Norm constraint Theta, such that the last period portfolio return (T-th one) is maximized. That is, solve the problem: min_w w'COVw$ s.t. w1+w2+...+wN=1 and |w1|+|w2|+...+|wN|<=Theta and, out of all values of Theta, choose the one with the maximum value of w'r_T, where r_T is the vector of assets returns in the last period and COV is the variance-covariance matrix of asset returns.

Description of the problem:

First I tried the "naive" approach: with a grid of 0.001 for Theta from 1 to 6 I was going to solve the portfolio optimization problem and compute the last period portfolio return for each Theta. The idea was to choose then the value of Theta with the largest corresponding last period return. However, I noticed that for quite a few values of Theta the solnp function did not converge. The problem occured mostly for small values of Theta: from 1 to 3. For larger values no problems with convergency were detected.

The second approach was to use solnp function twice: first as the function to find Theta and second as the inner part of the objective function. However, I could not find reliable estimates in this way: the values I got did not deliver the optimal solution. Apparently, the objective function is not smooth, but gosolnp function does not find the solution.

The code with data (6 assets with 120 return observations) is provided below. Any suggestions are welcome.

> exp_d
        [,1]     [,2]     [,3]     [,4]     [,5]     [,6]
1     1.3724   0.9081  -0.0695   5.7168   1.9642   1.4222
2     0.6095   1.5075   5.3842   2.7154   2.6838   6.3154
3    -2.6779  -0.1359  -0.4374   1.4287   0.0709  -0.7967
4    -3.5365  -4.3572  -2.0112  -3.5898  -2.3460  -4.0970
5     3.1210   3.6608   2.0944   3.1292   2.8965   3.4614
6     2.7364   1.8411   3.2639   2.9678   2.6067   2.3950
7    -1.0001  -0.3782   3.9316  -0.2621   0.0347   4.4635
8     3.9022   6.3784   6.6192   5.0044   3.5568   8.6305
9    -1.6000  -0.9889  -3.1676   1.3025   0.2071  -2.4764
10   -1.3184   0.8741   3.4796   3.0510  -0.7634  -0.5452
11    5.5482   3.3467  13.3256   5.4076   5.1017   7.4921
12   -1.5484   1.3040  -3.9474  -0.9628  -3.0156  -1.6326
13    4.3331   5.2347   3.9846   9.2369   6.7429   7.2603
14    2.3503  -2.2044   0.8600   3.9191   1.2181  -1.9651
15    2.3981   2.2316   0.3990   5.3864   4.3919   5.9674
16   -0.1633  -2.1458  -5.8357  -3.6349  -4.2840  -6.6219
17   10.4346   8.0620  10.2275   7.0560   6.7676   6.6346
18    5.5505   2.6016   2.4506   2.4954   1.8547   3.4755
19    3.2031   2.7804   3.5948  -0.4774  -0.3667  -2.3168
20   -4.7913  -1.7203  -4.1271  -0.6762  -1.1395  -2.7296
21    7.3930   8.6229   9.4570  12.2800   6.1327   7.8254
22    4.2158  10.6845   9.9723   2.9145   6.0000   4.4979
23    7.5326   1.9540   2.5740   2.6065  -0.1128   0.6388
24   -8.5131  -8.3044  -6.8294  -3.6094  -4.1224  -5.4164
25   -0.4048  -0.4017  -0.8867   1.3590   0.1098   0.9017
26    6.1240   5.0517   3.6990   8.7368   5.3867   6.9468
27    5.7317   4.4538   6.1762   3.4108   1.9153   4.4896
28    6.8299   3.1244   1.6621   1.3590   1.4325   2.0067
29    8.6705  11.1936  12.2831  11.1602  13.2781  13.1497
30    0.9055  -0.5953  -0.6462   0.9332  -0.0008   1.2917
31   -0.0340   1.9379   1.4480   6.5262   4.8373   2.6307
32    0.7414   1.1014   0.3820  -0.5791   0.8306   3.1476
33   -5.9533  -3.4602  -4.1597  -1.3835   2.2098  -0.0642
34   -0.0822   2.7549   0.5136   2.2172   1.1145   2.8362
35  -10.2009  -9.3603 -12.8907  -5.7297  -4.1622  -6.1709
36    6.9673   9.4356   6.4064  12.4593   9.3500   7.5052
37   -0.0681  -0.7029   3.8633   3.5471   6.2018   4.9529
38   -0.8885  -0.4641  -0.4544   7.2353  12.5430   6.9497
39   -4.2569  -4.3105  -3.6218  -4.4211  -4.9685  -6.3175
40  -21.7261 -19.3351 -19.8329 -24.1865 -14.0943 -10.3723
41  -16.4096  -9.8425 -12.3366 -14.2535 -10.9314  -7.6677
42   -2.7511  -2.7249  -1.8488   3.3988   0.5515   1.3170
43    6.2347   9.4395   8.4631   7.1735   3.3113   2.9221
44    1.3502   1.2075   4.3779   3.4639   1.8818   1.2221
45    9.0009  11.0661  10.7647   6.8160   8.3348   5.3573
46   -6.0956  -1.7406  -3.4814  -2.3610  -1.7939  -6.6523
47   -3.7611  -2.3132  -2.9578   0.7061  -2.2643  -1.3650
48  -16.9330 -16.6497 -18.9287 -17.8490 -13.1565 -11.8439
49    6.5208   3.7280   2.5406   4.4017   4.2198   5.2644
50   -3.7646  -1.6135  -2.5073   1.1969  -0.8000  -1.6232
51  -13.6823 -14.6721 -19.1619 -12.0325 -11.7394 -17.0858
52  -10.1007  -7.6801 -10.7443  -9.4278  -7.3264 -11.4835
53    0.2234  -2.7838  -2.6023  -2.3616  -2.8339  -6.6002
54  -10.4236 -11.0266 -17.2866  -5.8057  -9.1776  -9.7245
55    9.9121  11.3602  15.4627   4.7170   6.9823  13.5911
56   13.2032  11.6577  17.0900  12.8706   6.7357  12.1756
57   -4.3440  -2.9204  -6.9601  -4.1971 -10.3577  -8.8826
58  -12.8949 -14.0212 -18.2229  -8.8412 -11.1070 -11.3490
59   -9.6709  -8.7006 -13.8440 -11.3932 -15.9343 -20.3630
60   10.1262   7.7295  20.3174  12.8509  16.1956  25.2657
61   -8.5867  -8.9411  -5.7363  -5.1668 -10.1775 -12.2143
62   -1.6114  -1.1797  -3.7504   0.8986  -1.5948   0.0531
63  -26.5722 -30.0521 -33.8286 -28.8671 -28.1493 -35.1131
64    4.2319   9.2071   5.4593   9.5401   3.2588  11.7214
65   -7.7335  -7.2915  -9.3902  -6.1288 -16.6307 -14.5665
66  -13.7987 -16.5088 -22.4192 -11.5667 -18.8221 -20.6686
67    2.4924   3.9264  10.5116  -3.4347   1.9132   6.6353
68   -2.4754   1.9822   1.3833   7.1142   1.7366   0.3507
69  -10.5259 -12.1366 -10.6244 -10.2031 -15.1403 -14.7460
70  -19.3072 -17.1449 -16.0430 -18.1275 -20.1476 -18.5418
71  -17.1387 -24.8400 -17.4741 -19.2184 -25.9531 -25.4360
72    0.9540   6.2481   1.0364  -1.7110   0.0545   8.8476
73   31.9086  36.1125  63.2475  28.1955  48.6209  67.7847
74   48.7266  55.3725  83.5754  31.3211  50.9661  62.5277
75   -2.8054  -4.5841 -12.4737  -1.0366  -6.5149  -5.0087
76  -16.2535 -19.6450 -23.8672 -10.7514 -17.5126 -23.2272
77   -0.2972  -8.2786 -13.2053  -2.7223  -8.9288 -16.3816
78   -5.1581  -4.2529 -11.0631   6.8503   1.0438  -3.5633
79    4.3896   1.3914   7.8950  -0.0120   3.0297   8.7685
80  -17.8468 -18.4137 -20.8881 -14.9493 -16.5816 -17.3738
81    7.0790   6.1577  16.0332   0.9569   9.8608   6.7577
82   45.4453  54.8024  56.5610  33.7390  51.8933  57.4905
83   59.9378  62.1965  73.3394  16.9717  26.7781  41.6260
84   32.9373  23.3035  18.5882  11.3683  15.5089  21.9496
85  -14.1385 -12.7712  -7.2418  -8.8583 -13.0573  -9.2208
86   10.5585  10.1102   8.2144  10.5958  15.7784  18.8615
87   -7.6121 -13.3352 -20.9042  -9.1454 -12.3603 -19.3866
88   -8.4949 -12.4200 -13.8362  -6.1614 -10.5922 -17.7448
89    4.4492   4.6618   6.1086   9.6900  12.2772  12.6924
90    4.7702   4.0567   0.4565   2.1093   1.8286   3.3536
91   21.5203  28.5643  38.3403   9.8669  16.8246  24.1368
92   -0.2935   1.1796   5.3723  -2.3815  -3.1241  -3.8100
93    4.7722   2.0674  -0.0727  -0.0029  -0.1362  -0.6200
94    2.8356  -1.3170  -1.8303  -1.7780  -2.3152  -4.6160
95   -7.3381  -9.8907 -12.0504  -6.4033  -8.5879 -13.5106
96    3.7672   0.0811  -2.3100   2.6280   2.4811   2.8615
97  -18.0226 -21.5042 -24.4760  -8.3110 -11.7051 -23.1790
98   10.9235   9.4125  12.2230   6.0718   4.4464   5.7691
99   -0.9255  -0.5806  -3.5672  -0.2597  -0.1487  -0.1050
100  -0.9250  -1.6380  -4.1937  -0.4115  -2.7080  -7.3690
101  17.4218  15.4433  12.5284  10.2402   5.0137  10.6573
102   4.9409   1.7554   1.5981   0.7507   0.5333  -2.2372
103  -5.5523  -3.3988  -3.0325  -3.0099  -2.8838  -9.3128
104  -3.5225  -4.8589  -5.8780  -0.8581  -1.6838 -12.9875
105  -5.9234  -7.4692 -11.2668  -2.9715  -3.3289  -7.7499
106   6.7174   9.3484  10.2238   8.0311   9.8871  13.0598
107  -2.4801   2.3317   1.7345   3.1370   4.3390   3.9570
108  -1.7502   5.9262   0.8071   6.3821   5.6136   8.0813
109   9.1528  12.9358  12.7537   6.9882   6.3196  17.1148
110   4.2227   8.6796  14.4462   2.5694   2.2001   3.7475
111   5.0917   5.5331   0.5575   4.6757   0.6768   1.1198
112  10.9064  10.8504   6.8068   6.6016   7.9769   6.0328
113   6.6969  10.4475  18.9743   3.0389   5.6087  14.5739
114   5.7713  10.1556   2.2152   2.9835   5.8064   8.6858
115  10.3194   7.6727  22.3771   3.2881   9.5299  12.2406
116   1.9010   6.5601   6.7824   1.9137   2.8060   7.1039
117   0.5096   2.3380   0.8324   2.6378   0.0632  -1.0088
118 -14.3931 -13.9743 -15.4640  -7.1563  -8.3505 -10.2494
119   4.9011   5.5856   8.6767   5.0578   5.1107   6.5508
120  -2.2080  -0.2588  -1.2498   3.6325   2.1402   0.2676

#define equality constraint function 
equal <- function(x) c(sum(x))

#define inequality constraint function
in_inequal <- function(x) c(sum(abs(x)))

#define objective function1
obj_f <- function(x) {

  int_r <- t(x)%*%V_C_M%*%x
  c(as.numeric(int_r))

}

#define objective function2
ex_obj_f <- function(x) {

  tteta <- x

  port_w <- solnp(rep(1/n,n), fun = obj_f, eqfun=equal, eqB=1, 
       ineqfun = in_inequal, ineqLB = 0, ineqUB = tteta, control = list(trace=0))

  lp_ret <- exp_d[nrow(exp_d),]%*%port_w$pars

  -lp_ret

}

#First "naive" attempt
exp_d <- as.matrix(exp_d)
n <- 6
V_C_M <- cov(exp_d)

res <- matrix(0:0, nrow = 5000, ncol = 3)

for (i in 1:5000) {

  tteta <- 1 + i*0.001
  port_w <- solnp(rep(1/n,n), fun = obj_f, eqfun=equal, eqB=1,  ineqfun = in_inequal, ineqLB = 0, ineqUB = tteta, control = list(trace=0))
  lp_ret <- exp_d[nrow(exp_d),]%*%port_w$pars

  res[i,1] <- tteta
  res[i,2] <- lp_ret
  res[i,3] <- port_w$convergence

}

#Second Approach (the result really depends on the starting value of the parameter)
tt_op1 <- solnp(pars = 1.5, fun = ex_obj_f, LB = 1, UB = 10, control = list(trace=1))
tt_op2 <- gosolnp(pars = 1.5, fun = ex_obj_f, LB = 1, UB = 10)

P.S. I have read posts with similar problems here, but coud not find a solution to my question.

Answer 1

Your model can be formulated as a pure QP (quadratic programming) problem instead of a difficult nonlinear problem with a nonlinear non-differentiable constraint.

The constraint

 sum(i, |x(i)|) <= Theta

can be linearized in different ways. One possible reformulation is

 -y(i) <= x(i) <= y(i)
 sum(i, y(i)) <= Theta
 non-negative (or free) variable y(i)

Now you can solve the model with a QP solver instead of a general purpose NLP solver.