A model-free approach to high-dimensional inference