Some glycoside hydrolases have broad specificity for hydrolysis of glycosidic bonds, potentially increasing their functional utility and flexibility in physiological and industrial applications. To deepen the understanding of the structural and evolutionary driving forces underlying specificity patterns in glycoside hydrolase family 5, we quantitatively screened the activity of the catalytic core domains from subfamily 4 (GH5_4) and closely related enzymes on four substrates: lichenan, xylan, mannan, and xyloglucan. Phylogenetic analysis revealed that GH5_4 consists of three major clades, and one of these clades, referred to here as clade 3, displayed average specific activities of 4.2 and 1.2U/mg on lichenan and xylan, approximately 1 order of magnitude larger than the average for active enzymes in clades 1 and 2. Enzymes in clade 3 also more consistently met assay detection thresholds for reaction with all four substrates. We also identified a subfamily-wide positive correlation between lichenase and xylanase activities, as well as a weaker relationship between lichenase and xyloglucanase. To connect these results to structural features, we used the structure of CelE from Hungateiclostridium thermocellum (PDB 4IM4) as an example clade 3 enzyme with activities on all four substrates. Comparison of the sequence and structure of this enzyme with others throughout GH5_4 and neighboring subfamilies reveals at least two residues (H149 and W203) that are linked to strong activity across the substrates. Placing GH5_4 in context with other related subfamilies, we highlight several possibilities for the ongoing evolutionary specialization of GH5_4 enzymes.