Xu was born in Haining, Zhejiang and graduated from the famous Hangzhou High School (浙江省杭州高级中学).
In 1915, he married Zhang Youyi and next year he went to Peiyang University (Beiyang University, now Tianjin University) to study Law. In 1917, he transferred to Peking University due to the law department of Peiyang University merging into Peking University. In 1918, after studying at Peking University, he traveled to the United States to study history in Clark University. Shortly afterwards, he transferred to Columbia University in New York to study economics and politics in 1919.
Finding the States "intolerable", he left in 1921 to study at King's College, Cambridge in England, where he fell in love with English romantic poetry like that of Keats and Shelley, and was also influenced by the French romantic and symbolist poets, some of whose works he translated into Chinese. In 1922 he went back to China and became a leader of the modern poetry movement. In 1923, he founded the Crescent Moon Society.
When the Bengali poet Rabindranath Tagore visited China, Xu Zhimo played the part of oral interpreter. Xu was also renowned for his use of the vernacular language. He was one of the first Chinese writers to successfully naturalize Western romantic forms into modern Chinese poetry.
He worked as an editor and professor at several schools before dying in a plane crash on November 19, 1931 near Tai'an, Shandong while flying from Nanjing to Beijing. He left behind four collections of verse and several volumes of translations from various languages.