Abstract Graphical models are a powerful tool to estimate high-dimensional inverse covariance (precision) matrix, which has been applied for portfolio allocation problem. The assumption made by these is sparsity of the precision matrix. However, when stock returns driven common factors, such does not hold. We address this limitation and develop framework, Factor Lasso (FGL), integrates graphica...