Staff profile

🏠Homepage 🎓Google Scholar 🎞️YouTube 🔗LinkedIn 𝕏 Twitter 💻Github ✉️Email

Research Interests: Computer Vision, Computer Graphics, Machine Learning, Biomedical Engineering

Dr Hubert P. H. Shum is an Associate Professor in Visual Computing and the Deputy Director of Research of the Department of Computer Science at Durham University, specialised in Spatio-Temporal Modelling and Responsible AI. His research is fundamental to many disciplines, facilitating him to engage in various multidisciplinary research centres. He is a Co-Founder of the Responsible Space Innovation Centre, a Fellow of the Wolfson Research Institute for Health and Wellbeing, and a Steering Group Member of the Centre for Visual Arts and Culture. Before this, he was the Director of Research/Associate Professor/Senior Lecturer at Northumbria University, and a Postdoctoral Researcher at RIKEN Japan. He received his PhD degree from the University of Edinburgh, and his Master's and Bachelor's degrees from the City University of Hong Kong.

To develop his team, he has led research projects as the Principal Investigator awarded by EPSRC, the Ministry of Defence, the Royal Society and Innovate UK. He has also been a Co-Investigator in NortHFutures, a £4.17 mil EPSRC project to found a new digital health hub in the North East England. These have facilitated him to have supervised more than 30 PhDs, recruited multiple Post-doctoral Researchers, and collaborated with international researchers from the UK, China, France, Japan and India. To engage the academic and industrial networks, he has chaired international conferences such as Pacific Graphics, BMVC, ACM SIGGRAPH/Eurographics SCA and ACM SIGGRAPH MIG. Contributing to the research community, he has served as an Associate Editor for Computer Graphics Forum, a Guest Editor for the International Journal of Computer Vision, and a Program Committee member in 19 conferences such as Eurographics, CVPR and Pacific Graphics.

He has authored over 150 research publications in the fields of Computer Vision, Computer Graphics, Machine Learning and Biomedical Engineering, particularly focusing on modelling human-related data with deep learning.

Joining Us

If you aspire to develop yourself and conduct high-quality research, do explore working with the team as a PhD student, a visiting/internship student, or an academic visitor. Dr Shum's team has a supportive culture with members developing successful research and enjoying their vibrant lifes in the UK. See further information and application procedures.

Some self-funded PhD oppotunities include:

• Biomedical Engineering with Deep Learning based Video Analysis
• Computer Vision with Deep Learning for Human Data Modelling
• Deep Learning based Computer Graphics for Creating Virtual Characters

Recent Research

• Multi-Task Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising (IEEE Transactions on Visualization and Computer Graphics)

• HINT: High-quality INpainting Transformer with Mask-Aware Encoding and Enhanced Attention (IEEE Transactions on Multimedia)

• A Video-Based Augmented Reality System for Human-in-the-Loop Muscle Strength Assessment of Juvenile Dermatomyositis (IEEE Transactions on Visualization and Computer Graphics)

• Hierarchical Graph Convolutional Networks for Action Quality Assessment (IEEE Transactions on Circuits and Systems for Video Technology)

• Interaction-Aware Decision-Making for Automated Vehicles using Social Value Orientation (IEEE Transactions on Intelligent Vehicles)

• Less is More: Reducing Task and Model Complexity for 3D Point Cloud Semantic Segmentation (Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition)

• Hard No-Box Adversarial Attack on Skeleton-Based Human Action Recognition with Skeleton-Motion-Informed Gradient (Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision)

• Unaligned 2D to 3D Translation with Conditional Vector-Quantized Code Diffusion using Transformers (Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision)

• A Pose-Based Feature Fusion and Classification Framework for the Early Prediction of Cerebral Palsy in Infants (IEEE Transactions on Neural Systems and Rehabilitation Engineering)

• Geometric Features Informed Multi-Person Human-Object Interaction Recognition in Videos (Proceedings of the 2022 European Conference on Computer Vision)

• Pose-Based Tremor Classification for Parkinson’s Disease Diagnosis from Video (Proceedings of the 2022 International Conference on Medical Image Computing and Computer Assisted Intervention)

• 360 Depth Estimation in the Wild - The Depth360 Dataset and the SegFuse Network (Proceedings of the 2022 IEEE Conference on Virtual Reality and 3D User Interfaces)

Research Grants

• EPSRC Digital Health Hub Pilot Scheme (Ref: EP/X031012/1), "Northern Health Futures Hub (NortHFutures)", Co-investigator, £4.17 million
• Knowledge Transfer Partnership (Ref: 10050391), "Skin Lesion Classification with Real-World Data", Principal Investigator, £143,393
• Durham University Strategy Research Fund, "Durham University Responsible Space Innovation Centre", Co-Founder and Co-Investigator, Department of Computer Science Leader, £2.76 million
• Defence and Security Accelerator (Ref: DSTL0000007030, ACC6031106), "Autonomous Persistent Wide Area Surveillance of Human and Vehicle Activity", Co-investigator, £93,978
• Security Technology Research Innovation Grants Programme (Ref: 007CD), "Tracking Drones Across Different Platforms with Machine Vision", Principal Investigator
• Defence and Security Accelerator (Ref: DSTLX-1000140725), "D-FOCUS: Drone-FOrmation Control for countering future Unmanned aerial Systems", Principal Investigator, £124,901
• GX Project Grants Scheme, "Wound Monitoring with Depth Cameras on Portable Systems", Principal Investigator
• Royal Society International Exchanges (Ref: IES\R2\181024), "Modelling Human Motion for Synthesis and Recognition with Deep Learning on Surface Features", Principal Investigator
• Royal Society International Exchanges (Ref: IE160609), "An Affective Smart Environment for Personalized Learning and Teaching", Principal Investigator
• Erasmus Mundus Action 2 Programme (Ref: 2014-0861/001-001), "Sustainable Green Economies through Learning, Innovation, Networking and Knowledge Exchange (gLink)", Northumbria University Funding Management Officer, €3.03 million
• EPSRC (Ref: EP/M002632/1), "Interaction-based Human Motion Analysis", Principal Investigator, £123,819

Professional Activities

Conference Services

• Conference Chair of Pacific Conference on Computer Graphics and Applications 2022
• Conference Chair of ACM SIGGRAPH/Eurographics Symposium on Computer Animation 2022
• Conference Chair of the ACM SIGGRAPH Conference on Motion, Interaction and Games 2019
• Program Chair of the British Machine Vision Conference 2018
• Program Chair of the International Conference on Software, Knowledge, Information Management and Applications 2018
• Program Chair of the ACM SIGGRAPH Conference on Motion in Games 2016
• Area Chair of the British Machine Vision Conference 2023, 2022, 2021 and 2020

Journal Editorships

• Computer Graphics Forum (CGF), Associate Editor, 2019 - 2023
• International Journal of Computer Vision (IJCV), Guest Editor, Special Issue on Machine Vision and Deep Learning, 2020

Awards

• Exceptional Achievement for Excellence in PhD Supervision by Durham University in 2023
• Exceptional Achievement for Excellence in Teaching by Durham University in 2022
• Best Paper Award in the IEEE International Conference on Human-Machine Systems 2021
• Best Student Paper Award in the International Conference on Computer Graphics Theory and Applications (GRAPP) 2020
• Best Poster Award in the ACM SIGGRAPH Conference on Motion, Interaction and Games (MIG) 2019
• Best Paper Award in the International Conference on Advances in Computer Entertainment Technology (ACE) 2017
• Best Poster Award in the ACM SIGGGRAPH Conference on Motion in Games (MIG) 2016

Media Coverage

• Interviewed by the UK Catapult Network on how AI and computer vision enhance security, 2023

• 2000: Deputy Director of Research in the Department of Computer Science:

Conference Paper

• Liu, M., Frawley, J., Wyer, S., Shum, H. P. H., Uckelman, S. L., Black, S., & Willcocks, C. G. (in press). Self-Regulated Sample Diversity in Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics
• Lu, Z., Wang, H., Chang, Z., Yang, G., & Shum, H. P. (in press). Hard No-Box Adversarial Attack on Skeleton-Based Human Action Recognition with Skeleton-Motion-Informed Gradient.
• Li, B., Ho, E. S. L., Shum, H. P. H., & Wang, H. (in press). Two-Person Interaction Augmentation with Skeleton Priors.
• Corona-Figueroa, A., Shum, H. P. H., & Willcocks, C. G. (in press). Repeat and Concatenate: 2D to 3D Image Translation with 3D to 3D Generative Modeling.
• Liu, J., Yu, Z., Breckon, T. P., & Shum, H. P. H. (2024). U3DS3 : Unsupervised 3D Semantic Scene Segmentation. In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (3747-3756). https://doi.org/10.1109/WACV57701.2024.00372
• Crosato, L., Wei, C., Ho, E. S. L., Shum, H. P. H., & Sun, Y. (2024). A Virtual Reality Framework for Human-Driver Interaction Research: Safe and Cost-Effective Data Collection. In HRI '24: Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction (167-174). https://doi.org/10.1145/3610977.3634923
• Zhou, K., Chen, C., Ma, Y., Leng, Z., Shum, H. P., Li, F. W., & Liang, X. (2023). A Mixed Reality Training System for Hand-Object Interaction in Simulated Microgravity Environments. In 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). https://doi.org/10.1109/ISMAR59233.2023.00031
• Feng, Q., Shum, H. P., & Morishima, S. (2023). Enhancing Perception and Immersion in Pre-Captured Environments through Learning-Based Eye Height Adaptation. In 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). https://doi.org/10.1109/ISMAR59233.2023.00055
• Zhang, X., Zheng, S., Shum, H. P., Zhang, H., Song, N., Song, M., & Jia, H. (2023). Correlation-Distance Graph Learning for Treatment Response Prediction from rs-fMRI. In Neural Information Processing 30th International Conference, ICONIP 2023, Changsha, China, November 20–23, 2023, Proceedings, Part IX (298-312). https://doi.org/10.1007/978-981-99-8138-0_24
• Vijendran, M., Li, F. W., & Shum, H. P. (2023). Tackling Data Bias in Painting Classification with Style Transfer. In Proceedings of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 5 VISAPP: VISAPP (250-261). https://doi.org/10.5220/0011776600003417
• Chang, Z., Findlay, E. J., Zhang, H., & Shum, H. P. (2023). Unifying Human Motion Synthesis and Style Transfer with Denoising Diffusion Probabilistic Models. In Proceedings of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2023) - GRAPP (64-74). https://doi.org/10.5220/0011631000003417
• Li, L., Shum, H. P., & Breckon, T. P. (2023). Less is More: Reducing Task and Model Complexity for 3D Point Cloud Semantic Segmentation. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/CVPR52729.2023.00903
• Gaus, Y., Bhowmik, N., Issac-Medina, B., Atapour-Abarghouei, A., Shum, H., & Breckon, T. (2023). Region-based Appearance and Flow Characteristics for Anomaly Detection in Infrared Surveillance Imagery. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). https://doi.org/10.1109/CVPRW59228.2023.00301
• Corona-Figueroa, A., Bond-Taylor, S., Bhowmik, N., Gaus, Y. F. A., Breckon, T. P., Shum, H. P., & Willcocks, C. G. (2023). Unaligned 2D to 3D Translation with Conditional Vector-Quantized Code Diffusion using Transformers. In ICCV '23: Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision. https://doi.org/10.1109/ICCV51070.2023.01341
• Chang, Z., Koulieris, G. A., & Shum, H. P. (2022). 3D Reconstruction of Sculptures from Single Images via Unsupervised Domain Adaptation on Implicit Models. . https://doi.org/10.1145/3562939.3565632
• Li, R., Katsigiannis, S., & Shum, H. P. (2022). Multiclass-SGCN: Sparse Graph-based Trajectory Prediction with Agent Class Embedding. In 2022 IEEE International Conference on Image Processing (ICIP) Proceedings (2346-2350). https://doi.org/10.1109/icip46576.2022.9897644
• Chen, S., Atapour-Abarghouei, A., Kerby, J., Ho, E. S., Sainsbury, D. C., Butterworth, S., & Shum, H. P. (2022). A Feasibility Study on Image Inpainting for Non-cleft Lip Generation from Patients with Cleft Lip. . https://doi.org/10.1109/bhi56158.2022.9926917
• Zhang, X., Al Moubayed, N., & Shum, H. P. (2022). Towards Graph Representation Learning Based Surgical Workflow Anticipation. . https://doi.org/10.1109/bhi56158.2022.9926801
• Feng, Q., Shum, H. P., & Morishima, S. (2022). 360 Depth Estimation in the Wild - The Depth360 Dataset and the SegFuse Network. . https://doi.org/10.1109/vr51125.2022.00087
• Organisciak, D., Poyser, M., Alsehaim, A., Hu, S., Isaac-Medina, B. K., Breckon, T. P., & Shum, H. P. (2022). UAV-ReID: A Benchmark on Unmanned Aerial Vehicle Re-Identification in Video Imagery. . https://doi.org/10.5220/0010836600003124
• Zhang, X., Zhang, H., & Shum, H. P. (2022). Pose-based Tremor Classification for Parkinson’s Disease Diagnosis from Video. . https://doi.org/10.1007/978-3-031-16440-8_47
• Corona-Figueroa, A., Frawley, J., Bond-Taylor, S., Bethapudi, S., Shum, H. P., & Willcocks, C. G. (2022). MedNeRF: Medical Neural Radiance Fields for Reconstructing 3D-aware CT-Projections from a Single X-ray. . https://doi.org/10.1109/embc48229.2022.9871757
• Qiao, T., Men, Q., Li, F. W., Kubotani, Y., Morishima, S., & Shum, H. P. (2022). Geometric Features Informed Multi-person Human-object Interaction Recognition in Videos. . https://doi.org/10.1007/978-3-031-19772-7_28
• Li, L., Ismail, K. N., Shum, H. P., & Breckon, T. P. (2021). DurLAR: A High-Fidelity 128-Channel LiDAR Dataset with Panoramic Ambient and Reflectivity Imagery for Multi-Modal Autonomous Driving Applications. . https://doi.org/10.1109/3dv53792.2021.00130
• Isaac-Medina, B. K., Poyser, M., Organisciak, D., Willcocks, C. G., Breckon, T. P., & Shum, H. P. (2021). Unmanned Aerial Vehicle Visual Detection and Tracking using Deep Neural Networks: A Performance Benchmark. . https://doi.org/10.1109/iccvw54120.2021.00142
• Leng, Z., Chen, J., Shum, H. P., Li, F. W., & Liang, X. (2021). Stable Hand Pose Estimation under Tremor via Graph Neural Network. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR) (226-234). https://doi.org/10.1109/vr50410.2021.00044
• Zhou, K., Cheng, Z., Shum, H. P., Li, F. W., & Liang, X. (2021). STGAE: Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising. . https://doi.org/10.1109/ismar52148.2021.00018

Journal Article

• Chen, S., Atapour-Abarghouei, A., & Shum, H. P. H. (2024). HINT: High-quality INpainting Transformer with Mask-Aware Encoding and Enhanced Attention. IEEE Transactions on Multimedia, https://doi.org/10.1109/TMM.2024.3369897
• Constable, M. D., Shum, H. P. H., & Clark, S. (2024). Enhancing surgical performance in cardiothoracic surgery with innovations from computer vision and artificial intelligence: a narrative review. Journal of Cardiothoracic Surgery, 19(1), Article 94. https://doi.org/10.1186/s13019-024-02558-5
• Zhang, H., Ho, E. S. L., Zhang, X., Del Din, S., & Shum, H. P. H. (2024). Pose-based tremor type and level analysis for Parkinson’s disease from video. International Journal of Computer Assisted Radiology and Surgery, https://doi.org/10.1007/s11548-023-03052-4
• Crosato, L., Tian, K., Shum, H. P., Ho, E. S., Wang, Y., & Wei, C. (2023). Social Interaction‐Aware Dynamical Models and Decision‐Making for Autonomous Vehicles. Advanced Intelligent Systems, https://doi.org/10.1002/aisy.202300575
• Zhou, K., Shum, H. P., Li, F. W., & Liang, X. (2023). Multi-Task Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising. IEEE Transactions on Visualization and Computer Graphics, https://doi.org/10.1109/TVCG.2023.3337868
• Chen, S., Atapour-Abarghouei, A., Ho, E. S., & Shum, H. P. (2023). INCLG: Inpainting for Non-Cleft Lip Generation with a Multi-Task Image Processing Network. Software impacts, 17, Article 100517. https://doi.org/10.1016/j.simpa.2023.100517
• Zhou, K., Cai, R., Ma, Y., Tan, Q., Wang, X., Li, J., …Liang, X. (2023). A Video-Based Augmented Reality System for Human-in-the-Loop Muscle Strength Assessment of Juvenile Dermatomyositis. IEEE Transactions on Visualization and Computer Graphics, 29(5), 2456-2466. https://doi.org/10.1109/tvcg.2023.3247092
• Crosato, L., Shum, H. P., Ho, E. S., & Wei, C. (2023). Interaction-aware Decision-making for Automated Vehicles using Social Value Orientation. IEEE Transactions on Intelligent Vehicles, 8(2), 1339-1349. https://doi.org/10.1109/tiv.2022.3189836
• Zhou, K., Ma, Y., Shum, H. P., & Liang, X. (2023). Hierarchical Graph Convolutional Networks for Action Quality Assessment. IEEE Transactions on Circuits and Systems for Video Technology, https://doi.org/10.1109/TCSVT.2023.3281413
• McCay, K. D., Hu, P., Shum, H. P., Woo, W. L., Marcroft, C., Embleton, N. D., …Ho, E. S. (2022). A Pose-based Feature Fusion and Classification Framework for the Early Prediction of Cerebral Palsy in Infants. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 30, 8-19. https://doi.org/10.1109/tnsre.2021.3138185
• Men, Q., Ho, E. S., Shum, H. P., & Leung, H. (2021). A Quadruple Diffusion Convolutional Recurrent Network for Human Motion Prediction. IEEE Transactions on Circuits and Systems for Video Technology, 31(9), 3417-3432. https://doi.org/10.1109/tcsvt.2020.3038145
• Hu, S., Shum, H. P., Liang, X., Li, F. W., & Aslam, N. (2021). Facial reshaping operator for controllable face beautification. Expert Systems with Applications, 167, Article 114067. https://doi.org/10.1016/j.eswa.2020.114067
• Zhang, Z., Ma, Y., Li, Y., Li, F. W., Shum, H. P., Yang, B., …Liang, X. (2020). Cumuliform Cloud Formation Control using Parameter-Predicting Convolutional Neural Network. Graphical Models, 111, Article 101083. https://doi.org/10.1016/j.gmod.2020.101083
• Hu, S., Shum, H., Aslam, N., Li, F. W., & Liang, X. (2020). A Unified Deep Metric Representation for Mesh Saliency Detection and Non-rigid Shape Matching. IEEE Transactions on Multimedia, 22(9), 2278-2292. https://doi.org/10.1109/tmm.2019.2952983
• Hu, S., Liang, X., Shum, H. P., Li, F. W., & Aslam, N. (2020). Sparse Metric-based Mesh Saliency. Neurocomputing, 400, 11-23. https://doi.org/10.1016/j.neucom.2020.02.106
• Shen, Y., Yang, L., Ho, E. S., & Shum, H. P. (2020). Interaction-Based Human Activity Comparison. IEEE Transactions on Visualization and Computer Graphics, 26(8), 2620-2633. https://doi.org/10.1109/tvcg.2019.2893247
• Bhattacharya, M., Roy, S., Mistry, K., Shum, H. P., & Chattopadhyay, S. (2020). A Privacy-Preserving Efficient Location-Sharing Scheme for Mobile Online Social Network Applications. IEEE Access, 8, 221330 - 221351. https://doi.org/10.1109/ACCESS.2020.3043621
• Kar, A., Pramanik, S., Chakraborty, A., Bhattacharjee, D., Ho, E. S., & Shum, H. P. (2020). LMZMPM: Local Modified Zernike Moment Per-unit Mass for Robust Human Face Recognition. IEEE Transactions on Information Forensics and Security, 16, 495-509. https://doi.org/10.1109/tifs.2020.3015552
• Wang, H., Ho, E. S., Shum, H. P., & Zhu, Z. (2019). Spatio-temporal Manifold Learning for Human Motions via Long-horizon Modeling. IEEE Transactions on Visualization and Computer Graphics, https://doi.org/10.1109/tvcg.2019.2936810
• Rueangsirarak, W., Zhang, J., Aslam, N., Ho, E. S., & Shum, H. P. (2018). Automatic Musculoskeletal and Neurological Disorder Diagnosis With Relative Joint Displacement From Human Gait. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 26(12), 2387-2396. https://doi.org/10.1109/tnsre.2018.2880871
• Zhang, J., Shum, H. P., Han, J., & Shao, L. (2018). Action Recognition From Arbitrary Views Using Transferable Dictionary Learning. IEEE Transactions on Image Processing, 27(10), 4709-4723. https://doi.org/10.1109/tip.2018.2836323
• Zhang, L., Shum, H. P., & Shao, L. (2017). Manifold Regularized Experimental Design for Active Learning. IEEE Transactions on Image Processing, 26(2), 969-981. https://doi.org/10.1109/tip.2016.2635440
• Liu, Z., Zhou, L., Leung, H., & Shum, H. P. (2016). Kinect Posture Reconstruction Based on a Local Mixture of Gaussian Process Models. IEEE Transactions on Visualization and Computer Graphics, 22(11), 2437-2450. https://doi.org/10.1109/tvcg.2015.2510000
• Zhang, L., Shum, H., & Shao, L. (2016). Discriminative Semantic Subspace Analysis for Relevance Feedback. IEEE Transactions on Image Processing, 25(3), 1275-1287. https://doi.org/10.1109/tip.2016.2516947
• Henry, J., Shum, H. P., & Komura, T. (2014). Interactive Formation Control in Complex Environments. IEEE Transactions on Visualization and Computer Graphics, 20(2), 211-222. https://doi.org/10.1109/tvcg.2013.116
• Shum, H. P., Ho, E. S., Jiang, Y., & Takagi, S. (2013). Real-Time Posture Reconstruction for Microsoft Kinect. IEEE Transactions on Cybernetics, 43(5), 1357-1369. https://doi.org/10.1109/tcyb.2013.2275945
• Shum, H. P., Komura, T., & Yamazaki, S. (2012). Simulating Multiple Character Interactions with Collaborative and Adversarial Goals. IEEE Transactions on Visualization and Computer Graphics, 18(5), 741-752. https://doi.org/10.1109/tvcg.2010.257
• Shum, H. P., Komura, T., Shiraishi, M., & Yamazaki, S. (2008). Interaction patches for multi-character animation. ACM Transactions on Graphics, 27(5), Article 114. https://doi.org/10.1145/1409060.1409067