Shanjiang Tang
Assistant Professor

School of Computer Science& Technology
Tianjin University

Contact55-B517, CS, Beiyang Campus, JinNan District, Tianjin, China, 300350 [map]
Tel: +**936
tashj [at] tju [dot] edu [dot] cn
http://cs.tju.edu.cn/faculty/tangshanjiang


Bio | Publications |Projects |Code |CV |Other Stuff

About MeI am an Assistant Professor in the School of Computer Science& Technology,Tianjin University. I received my Ph.D degree from Nanyang Technological University in 2015. I received the B.Eng.and M.Sc. degrees from School ofSoftware Engineering andSchool of Computer Science& Technology at TianjinUniversity in 2008 and 2011, respectively.

My general research interests primarily focus on large-scale computing systems, big data, and cloud computing, withspecial emphasis on the resource management and job scheduling for Hadoop/YARN system.Specifically, I am interested in designing new scheduling and resource allocation algorithms,analyzing their performance, and implementing them in large-scale computing systems. I am also interested in problemsat the intersection of computing systems and economics.

News04/2015, our paper, entitled “Dynamic Job Ordering and Slot Configurations for MapReduce Workloads ” has been accepted to IEEE Transactions on Services Computing.

06/2014, our paper, entitled “DynamicMR: A Dynamic Slot Allocation Optimization Framework for MapReduce Clusters ” has been accepted to IEEE Transactions on Cloud Computing.

03/2014, our paper, entitled “Long-Term Resource Fairness: Towards Economic Fairness on Pay-as-you-use Computing Systems ” has been accepted to ICS 2014.

03/2014, new website goes online.

PublicationsJournal ArticlesShanjiang Tang, Bu-Sung Lee, and Bingsheng He, “Dynamic Job Ordering and Slot Configurations for MapReduce Workloads,” IEEE Transactions on Services Computing, 2015.

Shanjiang Tang, Bu-Sung Lee, and Bingsheng He, “DynamicMR: A Dynamic Slot Allocation Optimization Framework for MapReduce Clusters,” IEEE Transactions on Cloud Computing, 2014. [Supplementary Document][slides]

Shanjiang Tang, Ce Yu, Jizhou Sun, Bu-Sung Lee, Tao Zhang, Zheng Xu, and Huabei Wu, “EasyPDP: An Efficient Parallel Dynamic Programming Runtime System for Computational Biology,” IEEE Transactions on Parallel and Distributed System, 2012. [Supplementary Document][slides]

Conference and Workshop ProceedingsShanjiang Tang, Bu-Sung Lee, and Bingsheng He, “Towards Economic Fairness for Big Data Processing in Pay-as-you-go Cloud Computing,” in CloudCom 2014 (Ph.D. Consortium), Singapore, Dec 2014. [slides]

Shanjiang Tang, Bu-Sung Lee, Bingsheng He and Haikun Liu, ‘‘Long-Term Resource Fairness: Towards Economic Fairness on Pay-as-you-use Computing Systems,’’ to be presented in the 28th International Conference on Supercomputing (ICS), Munich, Germany, June 2014. [slides]

Shanjiang Tang, Bu-Sung Lee, and Bingsheng He, ‘‘Dynamic slot allocation technique for MapReduce clusters,’’ In IEEE Cluster 2013, Indiana , USA, Sept 2013. [slides]

Shanjiang Tang, Bu-Sung Lee, and Bingsheng He, ‘‘MROrder: Flexible Job Ordering Optimization for Online MapReduce Workloads, ’’In Euro-Par 2013, Aachen, Germany, Aug 2013. [slides]

Jun Du, Ce Yu, Jizhou Sun, Chao Sun, Shanjiang Tang, and Yanlong Yin, ‘‘EasyHPS: A Multilevel Hybrid Parallel System for Dynamic Programming. ’’ in IPDPS Workshops 2013, Boston, USA, May 2013.

Shanjiang Tang, Bu-Sung Lee, and Bingsheng He, ‘‘ Speedup for Multi-Level Parallel Computing,’’ in IPDPS Workshops 2012, Shanghai, China, May 2013. [slides]

Shanjiang Tang, Ce Yu, Bu-Sung Lee, Chao Sun, and Jizhou Sun, ‘‘ Adaptive Data Refinement for Parallel Dynamic Programming Applications,’’ in IPDPS Workshops 2012, Shanghai, China, May 2012. [slides]

Technical ReportsShanjiang Tang, Bu-Sung Lee, and Bingsheng He, “Economic Fairness for Resource Sharing in Pay-as-you-use Cloud Computing.” [Technical Report-09-2014]

Shanjiang Tang, Zhaojie, Niu, Bu-Sung Lee, and Bingsheng He, “Multi-Resource Fair Allocation in Pay-as-you-go Cloud Computing.” [Technical Report-07-2014]

PatentsChao Sun, Jizhou Sun, Shanjiang Tang, et al., Multi-level parallel programming method, CN**0, Nov 2010.

Ce Yu, Shanjiang Tang, Jizhou Sun, et al.,Parallel programming model system of DAG oriented data driving type application and realization method, CN**9, May 2010.

Chao Sun, Jizhou Sun, Shanjiang Tang, et al., Visual modeling and code skeleton generating method for supporting design of multinuclear parallel program, CN**1, Nov 2010.

Ce Yu, Jizhou Sun, Zhen Xu, Huabei Wu, Shizhong Liao, Xiaojing Meng, Shanjiang Tang, et al.,MPI parallel programming system based on visual modeling and automatic skeleton code generation method, CN**5, June 2009.

ProjectsI focus on systems and algorithms for large-scale data-intensive computing. My projects include:

MRYARN: Pay-as-you-go is a popular billing model based on users' resource usage in the cloud. A user's demand is often changing over time, indicating that it is difficult to keep the high resource utilization all the time for cost efficiency. Resource sharing is an effective approach for high resource utilization. In view of the heterogeneous resource demands of workloads in the cloud, multi-resource allocation fairness is a must for resource sharing in cloud computing. MRYARN is proposed for multi-resource fair allocation on the cloud. It ensures that each user in cloud computing can at least get the amount of total resources as that under the exclusively non-sharing environment in the long term. Moreover, MRYARN can guarantee that no users can get more amount of total allocated resources over time by lying their demands. Finally, MRYARN has a mechanism to discourage users to submit cost-inefficient workloads, especially when there are some idle resources they truly do not need.(homepage)(Technical Report-07-2014)

LTYARN: Life is not fair, but with a little help, existing large-scale data processing systems (e.g., YARN, Spark, Dryad) can be, ensuring resource sharing between users. However, past work on fair sharing considered memoryless fairness, an instantaneous fair share without historical information considered. When it comes to cloud computing (i.e., pay-as-you-use computing), it fails to satisfy the service-as-you-pay fairness (i.e., the total service that each user enjoys should be proportional to her payment) from a long-term view. Long-Term Resource Fairness (LTRF) generalizes max-min fairness for this case. LTYARN implements LTRF for YARN in cloud computing. (homepage) (demo) (ICS'14) (CloudCom'14)(Technical Report-09-2014)

DynamicMR: Hadoop MRv1 uses the slot-based resource model with the static configuration of map/reduce slots. Due to the pre-configuration of distinct map slots and reduce slots which are not fungible on slave nodes, slots can be severely under-utilized, which significantly degrades the performance. Although YARN was proposed to address this problem by giving a new resource model of 'container' that either map and reduce tasks can run on, we keep the slot-based model by proposing an alternative technique called Dynamic Hadoop Slot Allocation (DHSA). It relaxes the slot allocation constraint to allow slots to be reallocated to either map or reduce tasks depending on their needs. Our experiments show that it consistently outperforms YARN by about 2% ~ 9% for multiple jobs due to the ratio control mechanism of running map/reduce tasks. Second, the speculative execution can tackle the straggler problem, which has shown to improve the performance for a single job but at the expense of the cluster efficiency. In view of this, we propose Speculative Execution Performance Balancing (SEPB) to balance the performance tradeoff between a single job and a batch of jobs. Third, delay scheduling has shown to improve the data locality but at the cost of fairness. Alternatively, we propose a technique called Slot PreScheduling that can improve the data locality but with no impact on fairness. Finally, by combining these techniques together, we form a step-by-step slot allocation system called DynamicMR that can improve the performance of Hadoop MRv1 significantly while maintaining the fairness. (homepage) (TCC'14) (Cluster'13)

MROrder: In Hadoop MRv1, different job submission orders will bring significantly varied performance results. MROrder is an automated MapReduce job ordering optimizaton prototype system. It targets at the online MapReduce workloads where MapReduce jobs arrives over time for various perfomane metrics, such as makespan, total completion time. Users just need to input some simple arguments. For example, users need to designate the job ordering performance metric( e.g., makespan, total completion time). The MROrder then starts to perform job ordering optimization automatically for online MapReduce jobs, based on user's configuration. (homepage) (Euro-Par'13) (TSC'15)

EasyPDP: To tackle the growing volume of genomic data, an efficient and easily-programmed high performance computing system is needed. EasyPDP is a parallel dynamic programming runtime system and an abstract programming model for computational biology and scientific computing applications. As a runtime system, it handles low-level thread creating, mapping, resource management, and fault tolerance issues automatically regardless of the system characteristics or scale. As an programming model, it allows users to describe applications and specify concurrency from the high level, without concerns about the complex details of parallel programming. (homepage) (demo) (TPDS'12) (IPDPS Workshop'13) (IPDPS Workshop'12)

Open SourceAlmost all of my work is open source:

The MRYARN (Multi-Resource YARN for cloud computing) fair scheduler is now available at http://sourceforge.net/projects/mryarn/.

The LTYARN fair scheduler is now available at http://sourceforge.net/projects/ltyarn/.

The DynamicMR framework can be downloaded at http://sourceforge.net/projects/dynamicmr/, and its component Dynamic Hadoop Fair Scheduler (DHFS) can be found at sourceforge.

The MROrder prototype is open at http://sourceforge.net/projects/mrorder/.

The EasyPDP framework is available at hpclab.

OthersPersonal ActivitiesI had experiences on a number of programming contests since I was a undergraduate student. I participated in the ACM International Collegiate Programming Contest in 2005 and 2006, and won 'Golden Prize' in the 31th ACM/ICPC Asia Programming Contest VS Google Cup on Programming Invitational Tournament of National College Students.I've now stopped doing contests, but I still love algorithmic and mathematical problems.
During my master study in 2009, I had internship at IBM CRL over four months during the summer holiday, where it germinates my strong interest on research.
I enjoy reading, swimming, fitness and playing badminton.

Professional ActivitiesReviewer for

TPDS: 2013, 2014

TCC: 2014, 2015

TSC: 2014, 2015

VLDB: 2014, 2015

HPDC: 2014

PPOPP: 2013

SOCC: 2013
Interesting LinksHow to Have a Bad Career in Research/Academia, by Prof. David A. Patterson

The Most Common Mistakes, by Ben Yagoda

How to Read a Paper, by Prof. S. Keshav

Write a Rapid Prototype First, by Prof. Terence Tao

Last updated on April-13, 2015.