I think the question is more involved. 5-6 core libraries are needed, yes. But look at Scala which had those libraries, got lots of hype, and yet never really took off in the mainstream like people thought it would.
There are good linear algebra libs (e.g. NumPy -> ojAlgo, ND4J, Koma), data frame (Pandas -> TableSaw, Krangl), and deep learning (Smile, DL4J).
But I think there are more factors beyond libraries.