TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling
EuroSys 2026
I am a master's student in Computer Science at Shanghai Jiao Tong University, advised by Prof. Fan Wu and Prof. Shengzhong Liu in NNE-Lab.
My research focuses on efficient AI, LLM post-training and inference, reinforcement learning systems, and machine learning systems.
- Efficient AI
- LLM Post Train
- LLM Inference
- RL System
- Machine Learning System
News
- 🎉 Our SmartThinker has been accepted by ICML 2026!
- 🎉 Our StructureBench has been accepted by IJCAI 2026!
- 🎉 Our C²KV has been accepted by KDD 2026!
- 🏆 Honored to receive the SenseTime Rising Star Intern Award!
- 🎉 Our TokenFlow has been accepted by EuroSys 2026!
- 🏆 Grateful to receive the National Scholarship for graduate students!
- 🏆 Our Pre³ received an Outstanding Paper Award at ACL 2025!
- 🎉 Our Pre³ has been accepted to ACL 2025 as an oral presentation!
Publications
Switch between selected cards and the complete paper list
Pre³: Enabling Deterministic Pushdown Automata for Faster Structured LLM Generation
ACL 2025 Oral · 218/1,699 (12.8%) 🏆 Outstanding Paper · Top 2.5%
TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling
EuroSys 2026
Pre³: Enabling Deterministic Pushdown Automata for Faster Structured LLM Generation
ACL 2025 Oral · 218/1,699 (12.8%) 🏆 Outstanding Paper · Top 2.5%
Open-Source Projects
LightLLM
A Python-based LLM inference and serving framework for high-throughput, low-latency deployment, with distributed serving and an efficient runtime architecture.
- Python
- LLM Serving
- Distributed Systems
Relax
An asynchronous reinforcement learning engine for omni-modal post-training at scale, with service-oriented orchestration and decoupled training and inference built on Ray Serve, Megatron-LM, and SGLang.
- Reinforcement Learning
- Distributed Systems
- LLM Infrastructure
Invited Talks
Huawei KADC 2025 · High-Performance Inference and Technology Innovation Forum
Invited talk: “Cloud–Edge Collaborative Inference Acceleration for Large Language Models” (大语言模型端云协同推理加速方案), at the Kunpeng Ascend Developer Conference 2025 in Beijing.
AI Time · ACL 2025 Outstanding Paper Sharing
Presented Pre³ and discussed how deterministic pushdown automata enable faster structured LLM generation.
Awards
- SenseTime Rising Star Intern Award
- National Scholarship (Graduate), China
- ACL Outstanding Paper Award
- Beijing Outstanding Graduate
- Beihang University Outstanding Graduate
- National Scholarship (Undergraduate), China
- National Scholarship (Undergraduate), China
Education
Shanghai Jiao Tong University
M.S. in Computer Science · NNE-Lab
Advised by Prof. Fan Wu and Prof. Shengzhong Liu. GPA: 3.95/4.0.
Beihang University
B.S. in Computer Science · Shen Yuan Honors College
Top 5% honors program. GPA: 3.92/4.0; rank: 1/50.
Experience
SenseTime Research
Research Intern · Large Model Systems and Toolchain Team
Designed buffer-aware preemptive scheduling for responsive LLM serving, developed grammar-constrained structured decoding, and explored confidence-based dynamic speculative decoding for multi-token prediction.