Machine reading extracts valuable knowledge from text (e.g., in the form of entities and their relationships) and stores these in a knowledge base or knowledge graph for further usage. This is a task of continual reading and learning based on the texts and the knowledge already acquired. In this project we port this task to a deep learning framework, where fundamental research is needed to represent both the unstructured text and the structured knowledge in an integrated way. This is challenging also because the extraction is performed with limited supervision of annotated training data. While continually learning and updating the extracted knowledge, the model has to avoid acquiring incorrect knowledge. We evaluate the continual learning models with two use cases: extraction of biomedical knowledge from a publication feed of biomedical documents, and extraction of facts from a newsfeed.