Watching the pitiful Sargent copy his sums in Nestor, Stephen thinks of how mothers have nurtured both this weak boy and himself: "Amor matris: subjective and objective genitive." The phrase means "love of mother" in Latin, but in two possible senses: the mother's love of her child, or the child's love of its mother.
Genitive is the possessive case in Latin, usually translated into English with the word "of" or with an apostrophe + "s." Here matris could mean "of the mother" in the "subjective" sense that the "of" attaches to her, the subject feeling the love; or it could mean "of the mother" in the "objective" sense that the "of" attaches to her, the object for whom the child feels love.
Both meanings concern Stephen, of course, because he is conscious of how abundantly his mother loved him and how poorly he repaid her love. However much distaste he may feel for Sargent and his mother, and for his own beginnings, he recognizes how crucially his happiness depends upon this love.