Ryder was born on March 8, 1877 at Oberlin, Ohio in the United States. He had his early education at Ann Arbor, Michigan and the Phillips Academy in Andover, Massachusetts, from which he graduated in June 1894, to join Harvard University. He got his A.B. degree from Harvard in June 1897. After teaching Latin and literature at Andover for a year, he went to Germany for graduate studies. He studied at the University of Berlin and the University of Leipzig, from which he got the degree of Doctor of Philosophy in 1901, with a dissertation on the Rbhus in the Ṛgveda. He was an instructor in Sanskrit at Harvard University from 1902 until January 1906, when he moved to the University of California at Berkeley, to its linguistics department, as an instructor in Sanskrit and German. He became an Instructor in Sanskrit alone later in the same year, became Assistant Professor in 1908, Associate Professor in 1919, and Professor in 1925. From his arrival at Berkeley until his death, Sanskrit was a separate department with Ryder as chairman and sole member, after which it was absorbed into the Department of Classics. When in summer 1920 Berkeley first began offering courses in religion and religious education, Ryder was among the faculty, along with Herbert Francis Evans, Walter Goodnow Everett, Morris Jastrow, Jr., and Walter B. Pitkin.

He was a member of the American Oriental Society and the American Philological Association. He was also a member of the Phi Beta Kappa Society, and wrote the society's annual poem for its meeting in 1912. It is also said that he was at one time ranked one of the two best chess players on the Pacific Coast.