Songyang Zhang

2795 Augustine Dr

Santa Clara, CA, USA

I am Songyang Zhang (张宋扬 - in Chinese, Pronounce). I am an Applied Scientist at Amazon Artificial General Intelligence team, building video generative models. I received Ph.D. from University of Rochester, advised by Prof. Jiebo Luo. Before that, I got my master’s degree from Zhejiang University advised by Prof. Jun Xiao and my bachelor’s degree from Southeast University. I’ve received the NAACL 2021 Best Long Paper Award. My research is on computer vision and natural language processing, especially the intersection between video and language.

[Resume]

News

Feb 3, 2024	One paper accepted at TMM.
Jul 31, 2023	I joined Amazon as an Applied Scientist.
Jun 29, 2023	I have successfully defended my phd thesis.
Jan 20, 2023	One paper is accepted by ICLR 2023.
Oct 14, 2022	Invited talk at UM-IoS Workshop @ EMNLP 2022.

Selected Publications

* denotes equal contribution. The full list is here.

Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation

Jie An*, Songyang Zhang*, Harry Yang, Sonal Gupta, Jia-Bin Huang, Jiebo Luo, and Xi Yin

In Submission, 2023

PDF Website

Learning a Grammar Inducer by Watching Millions of Instructional YouTube Videos

Songyang Zhang, Linfeng Song, Lifeng Jin, Haitao Mi, Kun Xu, Dong Yu, and Jiebo Luo

In Conference on Empirical Methods in Natural Language Processing, 2022

EMNLPOral

Bib PDF Code Slides Talk

@inproceedings{zhang2022training,
  title = {Learning a Grammar Inducer by Watching Millions of Instructional YouTube Videos},
  author = {Zhang, Songyang and Song, Linfeng and Jin, Lifeng and Mi, Haitao and Xu, Kun and Yu, Dong and Luo, Jiebo},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022}
}

Make-A-Video: Text-to-video Generation without Text-Video Data

Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman

In International Conference on Learning Representations, 2022

ICLR

Bib PDF Website News

@inproceedings{singer2022make,
  title = {Make-A-Video: Text-to-video Generation without Text-Video Data},
  author = {Singer, Uriel and Polyak, Adam and Hayes, Thomas and Yin, Xi and An, Jie and Zhang, Songyang and Hu, Qiyuan and Yang, Harry and Ashual, Oron and Gafni, Oran and Parikh, Devi and Gupta, Sonal and Taigman, Yaniv},
  booktitle = {International Conference on Learning Representations},
  year = {2022}
}

MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration

Thomas Hayes*, Songyang Zhang*, Xi Yin, Guan Pang, Sasha Sheng, Harry Yang, Songwei Ge, Qiyuan Hu, and Devi Parikh

In European Conference on Computer Vision, 2022

ECCV

Bib PDF Supp Code Slides Website Talk News

@inproceedings{hayes2022mugen,
  title = {MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration},
  author = {Hayes*, Thomas and Zhang*, Songyang and Yin, Xi and Pang, Guan and Sheng, Sasha and Yang, Harry and Ge, Songwei and Hu, Qiyuan and Parikh, Devi},
  booktitle = {European Conference on Computer Vision},
  year = {2022}
}

Expanding Language-Image Pretrained Models for General Video Recognition

Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling

In European Conference on Computer Vision, 2022

ECCVOral

Bib PDF Code

@inproceedings{ni2022expanding,
  title = {Expanding Language-Image Pretrained Models for General Video Recognition},
  author = {Ni, Bolin and Peng, Houwen and Chen, Minghao and Zhang, Songyang and Meng, Gaofeng and Fu, Jianlong and Xiang, Shiming and Ling, Haibin},
  booktitle = {European Conference on Computer Vision},
  year = {2022}
}

SAT: 2D Semantics Assisted Training for 3D Visual Grounding

Zhengyuan Yang, Songyang Zhang, Liwei Wang, and Jiebo Luo

In IEEE International Conference on Computer Vision, 2021

ICCVOral

Bib PDF Code

@inproceedings{yang2021sau,
  title = {SAT: 2D Semantics Assisted Training for 3D Visual Grounding},
  author = {Yang, Zhengyuan and Zhang, Songyang and Wang, Liwei and Luo, Jiebo},
  booktitle = {IEEE International Conference on Computer Vision},
  year = {2021}
}

Video-aided Unsupervised Grammar Induction

Songyang Zhang, Linfeng Song, Lifeng Jin, Kun Xu, Dong Yu, and Jiebo Luo

In Conference of the North American Chapter of the Association for Computational Linguistics, 2021

NAACLBest Long Paper Award

Bib PDF Code Poster Slides Talk News

@inproceedings{zhang2021video,
  author = {Zhang, Songyang and Song, Linfeng and Jin, Lifeng and Xu, Kun and Yu, Dong and Luo, Jiebo},
  title = {Video-aided Unsupervised Grammar Induction},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021}
}

Multi-Scale 2D Temporal Adjacency Networks for Moment Localization with Natural Language

Songyang Zhang, Houwen Peng, Jianlong Fu, Yijuan Lu, and Jiebo Luo

IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021

TPAMI

Bib PDF Code
@article{zhang2020multi, author = {Zhang, Songyang and Peng, Houwen and Fu, Jianlong and Lu, Yijuan and Luo, Jiebo}, journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence}, year = {2021} }
Learning Sparse 2D Temporal Adjacent Networks for Temporal Action Localization

Songyang Zhang, Houwen Peng, Le Yang, Jianlong Fu, and Jiebo Luo

arXiv preprint arXiv:1912.03612, 2019

arXivWinner of HACS Temporal Action Localization Challenge at ICCV 2019

Bib PDF Website
@article{zhang2019learning, author = {Zhang, Songyang and Peng, Houwen and Yang, Le and Fu, Jianlong and Luo, Jiebo}, journal = {arXiv preprint arXiv:1912.03612}, year = {2019} }
Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language

Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo

In the AAAI Conference on Artificial Intelligence, 2020

AAAI

Bib PDF Code Poster Slides News
@inproceedings{zhang2020learning, author = {Zhang, Songyang and Peng, Houwen and Fu, Jianlong and Luo, Jiebo}, booktitle = {the AAAI Conference on Artificial Intelligence}, year = {2020} }
Exploiting Temporal Relationships in Video Moment Localization with Natural Language

Songyang Zhang, Jinsong Su, and Jiebo Luo

In ACM International Conference on Multimedia, 2019

ACMMM

Bib PDF Code Poster Slides
@inproceedings{zhang2019exploiting, author = {Zhang, Songyang and Su, Jinsong and Luo, Jiebo}, booktitle = {ACM International Conference on Multimedia}, year = {2019} }

Fusing Geometric Features for Skeleton-Based Action Recognition Using Multilayer LSTM Networks

Songyang Zhang, Yang Yang, Jun Xiao, Xiaoming Liu, Yi Yang, Di Xie, and Yueting Zhuang

IEEE Transactions on Multimedia, 2018

TMM

Bib PDF

@article{zhang2018fusing,
  title = {Fusing Geometric Features for Skeleton-Based Action Recognition Using Multilayer LSTM Networks},
  author = {Zhang, Songyang and Yang, Yang and Xiao, Jun and Liu, Xiaoming and Yang, Yi and Xie, Di and Zhuang, Yueting},
  journal = {IEEE Transactions on Multimedia},
  year = {2018}
}

On Geometric Features for Skeleton-Based Action Recognition Using Multilayer LSTM Networks

Songyang Zhang, Xiaoming Liu, and Jun Xiao

In IEEE Winter Conference on Applications of Computer Vision, 2017

WACV

Bib PDF Code Poster Slides

@inproceedings{zhang2017geometric,
  title = {On Geometric Features for Skeleton-Based Action Recognition Using Multilayer LSTM Networks},
  author = {Zhang, Songyang and Liu, Xiaoming and Xiao, Jun},
  booktitle = {IEEE Winter Conference on Applications of Computer Vision},
  year = {2017}
}