The immediate adoption of deep learning models into domain-specific tasks for edge intelligence-based services still poses several challenges to overcome. The first is efficiently constructing the most suitable neural network architecture amongst the numerous types of available architectures. Once addressing this challenge, the second is understanding how to gather knowledge to build efficient neural network models from the user’s devices (i.e., smartphone) without affecting the user’s privacy. And the third critical issue is minimizing the gap between estimated and actual performance while building models. In this work, we propose a novel framework for deploying deep learning models for domain-specific tasks called MetaFed, which combines population-based meta-learning and federated learning to resolve the three challenges mentioned earlier. MetaFed autonomously constructs the potential domain-specific models with the help of population-based meta-learning by utilizing the model knowledge base. We improve the model knowledge base by using the population’s knowledge via federated learning while minimizing the performance estimation gap for model construction. Experimental results show that our proposed framework offers a promising solution for deploying deep learning models at edge devices, with a significant performance gain than the existing alternative approaches.