We aimed to evaluate the usefulness of current dialogue dataset annotation and propose annotation unification and automatized enhancements for better user modeling by training on larger amounts of data. Current datasets’ annotationis often only focused on annotation geared toward the dialog system learning how to answer, while the user representation should be explicit,consistent and as complete as possible for more complex user representation (e.g. cognitively).  Theproject will start from existing annotated dialog corpora and produce extended versions,  with improved annotation consistency and extra user representation annotations produced automatically from existing corpora like  bAbI++ and  MultiWOZ and others. We will explore unifying annotations from multiple datasets and evaluate the enhanced annotation using our own end-to-end dialogue models based on memorynetworks.