Provost & Academic Vice President
2005 - 2006 Graduate Program Assessment Summaries
College of Arts and Sciences
of African American Studies
Department of Biology
Department of Chemistry
Department of English & Journalism - English
Department of Geography
Department of History
Department of Mathematics
Department of Physics
Department of Political Science
Department of Psychology: Clinical/Community Mental Health; General Experimental; Specialist School Psychology
Department of Sociology
Department of Women's Studies
The purpose of this report is to describe the assessment plans as well as the current results of major assessment for 2005-2006. Department results are reported in alphabetical order.
The African American Studies department is in the process of developing its plan of assessment for the graduate certificate program.
- Knowledge Base: Students will develop an adequate knowledge base in eight core areas of biology: biometrics, ecology, genetics, evolution, physiology, molecular biology, organismal biology and systematics;
- Research: Students will exhibit the ability to function in a biological research setting; and
- Success of Graduates: Graduates of the program will demonstrate competency as biologists.
Direct and Indirect Measurement:
- Knowledge Base: Knowledge base is assessed by a final oral examination that is the capstone experience of the program. Non-thesis students also take an additional written comprehensive examination. The performance on the oral examination is evaluated by the student’s examination committee. The final oral examination is approximately 2 hours and contains questions that test the student’s mastery of the thesis or advanced project, the student’s knowledge in their specialization, and in the core areas of biology. The committee completes a written report (Final Oral Examination Report Form) evaluating the student’s performance in each of the above areas. The report is given to the Graduate Committee Chair to be added to the assessment profile;
- Research: The ability of students to function in a research setting is assessed by having students do research on a topic in biology, present a seminar on their results, and complete the writing of a thesis or advanced project report. The seminar and thesis or advanced project are evaluated by the student’s examination committee and graded as satisfactory or unsatisfactory (Biol 600, Biol 601). They are also asked questions concerning their thesis at the final oral examination;
- Success of Graduates: Graduates of the program must demonstrate competency as biologists as exhibited by obtaining employment or entering doctoral programs or professional schools. Alumni surveys on employment and acceptance to doctoral/professional programs will be used to assess the success of graduates. The data from surveys will be compiled and evaluated.
Results: Since May 6, 1998, 105 students have completed final oral examinations. Student performance in the assessed areas is summarized in the attached tables. All students assessed during this period successfully completed their theses or advanced projects. When tested on their area of specialization (botany, microbiology, zoology, or biology education), 95% of students exceeded or met our programmatic expectations. In the core areas of biology, the results were as follows: cell/molecular (96% exceeded or met), organismal (97% exceeded or met) and population/community (96% exceeded or met).
Based on our survey of graduates, most are employed and actively working as professional biologists or related professions (83%) and 17 students are attended or have completed graduate or professional schools (16%). The results for May 2005 to May 2006 are similar to the overall data.
Based on the data collected so far, the graduate students in the program are meeting our programmatic expectations. They are passing their thesis/advance project defenses (100%) and meeting or exceeding our programmatic expectations in their specialty area (95%) and most general areas of biology (96-97% on first oral exam, 100% after retesting). Essentially all graduates find jobs in areas related to their training or are able to go on to doctoral programs or professional schools.
Feedback Loop: Our goal is to continue with our current methods. During the past three academic years, we evaluated all our undergraduate and graduate core courses to make sure they are serving the needs of our students. Biol 502 was renamed (Molecular Applications in Organismal Biology) and has a new course description. Biol 503 also has a new course description. These changes reflect the evolution of these courses to meet the changing needs of our students.
Learning Outcomes: Students will
- demonstrate a basic knowledge of concepts and theories of the field (organic, bio, analytical, or physical chemistry);
- demonstrate a command of scholarly style;
- develop ability to define research objectives effectively;
- demonstrate ability to perform a study with appropriate methods/tools;
- acquire necessary analytical skills;
- develop acquired skills for independent research;
- demonstrate ability to communicate effectively;
- teaching assistants will demonstrate ability to teach.
Direct and Indirect Measurement:
- pre-test/post-test (American Chemical Society standardized exams) for learning outcomes 1 and 2;
- a comprehensive exam in the student’s area of interest is given two months prior to the oral defense of the master’s thesis/internship report;
- Learning outcomes 3-7 were evaluated through by a grade of A or B as “having met” in Chem 580 seminars (at least two);
- Learning outcome 8 was measured by evaluating laboratory teaching skills using a standardized form.
Results: There is pre-test data on 24 students. Seven of those students have graduated; therefore the comprehensive exam data is for seven students. The department reports that all seven met or exceeded expectations. The data obtained for all graduate teaching assistants (i.e. 15 in Fall 04 and 17 in Spring 05) suggested a wide performance range “very average” to “very strong or excellent” for overall effectiveness. Results of alumni surveys will be reported next year. Between 60 and 80 percent of graduates have been accepted by major Ph.D. schools nationwide.
Feedback Loop: no changes.
English In 2005-2006 the graduate program in the Department of English has been under thorough review. Assessment reporting will resume as usual for the 2006-2007 assessment cycle.
1) Geography Knowledge and Understanding. This is evident when the student is able to:
- Scientifically define spatial/geographic problems
- Understand sustainability principles for shaping the built environment
- Understand the social, economic, political structures in places and how these structures influence life in urban and rural areas
- Understand the nexus between people and the environment
- Understand the diversity and complexity of regions
2) Cognitive Characteristics. This is evident when the student is able to:
- Engage in a sustained, critical discourse about geographical theories
- Generate primary data and collect secondary data
- Critically analyze quantitative or collective data
- Collaborate with others
- Successfully design and conduct independent research
- Synthesize ideas from multiple sources
- Solve realistic problems using geographical concepts & principles
3) Geographical Methods. The student will effectively use one or more geographical methods to assess, formulate and resolve real world problems. These methods include:
- GIS for the apprehension and analysis of complex, multivariable problems conceptualized in field or object manner
- Deterministic or probabilistic quantitative methods for solving problems with spatial data (basic statistics, sample design, basic optimization, multivariable linear statistics, elementary spatial statistics, spatial correlation)
- Computerized cartographic methods (map production from base variables, use of appropriate projection, scale)
- Field methods (neighborhood, geomorphic, hydrologic surveys)
- Qualitative methods (text analysis)
- Meteorological methods of climate analysis and weather prediction
4) Practical/Professional Skills. This is evident when the student is able to:
- Effectively communicate research orally
- Write in a neat, clear and concise manner
- Effectively work within a group to set and accomplish tasks
- Independently conceptualize, execute and report research projects
- Create and interpret statistical information: tables, graphs, maps
- Use computer hardware and software effectively to accomplish research tasks
- Perceive and uphold high standards of morality and ethics in all aspects of professional behavior
- Demonstrate leadership qualities (in projects, clubs, internships)
- Identify the open questions over which scientific controversy exists, as opportunities for further research and knowledge advancement
Direct and Indirect Measurement:
Portfolio Assessments: Portfolios assess student learning for five degree streams. Each portfolio assesses:
- Communication Skills;
- Analytical Thinking and Reasoning Skills;
- Complexity Skills; and
- Scholarship Skills.
Portfolios should contain evidence of: 1) Knowledge-based skills and 2) Analysis-based skills.
Survey Questionnaires. All department majors are surveyed at least twice while enrolled in program. Students are asked to agree/disagree with statements on questionnaire. Continued contact will be maintained with the graduates from the department’s degree programs via regular (annual) surveys.
Results: New plan presented without data; full expectation of data 2006-2007.
Feedback Loop: None.
- achieve and demonstrate a basic and advanced level of knowledge in their basic field of study;
- achieve and demonstrate a basic and advanced level of knowledge in their secondary field of study;
- successfully demonstrate the ability to communicate effectively in written and oral presentations;
- successfully demonstrate the ability to analyze and interpret historical sources, data, and literature.
Direct and Indirect Measurement: At the final Oral Defense each member of the examining committee assesses each candidate on the student learning outcomes.
Results: During the 2005-2006 year nine students receiving the advanced degree met or exceeded expectations in the above listed skills and attributes.
Feedback Loop: no significant changes implemented.
Students will demonstrate:
- a satisfactory knowledge of the material in the core courses;
- knowledge in depth of a subject chosen from one of the following general areas of concentration: 1) pure mathematics; 2) computational and applied mathematics; 3) statistics; 4) an approved focus area;
- the use of abstract mathematical reasoning in the mathematics courses that they take in their first year of enrollment in the program;
- the ability to read, write, and present mathematics in the mathematics courses that they take in their fourth semester of full-time studies or equivalent.
Direct and Indirect Measurement:
Learning Outcome 1) All professors from whom the student takes core courses during the first year will be asked to assess the student’s knowledge of the core course material, rating it on a scale of 1 to 5, where “5” means excellent, “4” means good, “3” means fair, “2” means poor, and “1” means unacceptable. Short comments explaining the numerical rating will also be given, with a sample of the student’s work attached. Criteria for Success: In each of the four core courses, no student will be rated below 2, and at least 75% will be rated 3 or higher.
Learning Outcome 2) Any student who opts to earn a degree under Plan 1 of our program and takes an oral examination over advanced coursework, or any student who opts to earn a degree under Plan 2 of our program and writes a thesis, the student’s advisor and examining committee will be asked to assess the student’s understanding of the course material, rating it on a scale of 1 to 5, where “5” means excellent, “4” means good, “3” means fair, “2” means poor, and “1” means unacceptable. Criterion for Success: No student will be rated below 3, and at least 75% will be rated 4 or 5.
Learning Outcome 3) All professors from whom the student takes classes during the first year will be asked to assess the student’s ability to reason abstractly in the course of doing mathematics, rating it on a scale of 1 to 5, where “5” means excellent, “4” means good, “3” means fair, “2” means poor, and “1” means unacceptable. Criteria for Success: No student will be rated below 2, and at least 75% will be rated 3 or higher.
Learning Outcome 4) During a student’s fourth semester of full-time studies (or equivalent, if the student is enrolled part-time), all professors from whom the student takes classes will be asked to assess the student’s ability to read, write, and present mathematics. Reading, writing, and presenting will be assessed separately. Each will be rated on a scale of 1 to 5, where “5” means excellent, “4” means good, “3” means fair, “2” means poor, and “1” means unacceptable. Criteria for Success: In each of the three skills assessed -- reading, writing, and presenting mathematics -- no student will be rated below 2, and at least 75% will be rated 3 or higher.
Results: Grade distribution data submitted as indirect measurement.
Feedback Loop: Graduate plan under revision.
Physics has identified three learning outcomes for assessment of graduate student learning:
- demonstration of mastery of basic core theoretical physics,
- demonstration of competency in one or more areas of experimental physics, and
- demonstration of competency in one or more areas of computational physics.
Direct and Indirect Measurement: Assessment methods include entrance (interview, external data, and entrance examination), mid-program (monitored advising sessions), exit (internal data, any new external data, exit interview, and exit examination), and alumni assessment procedures (performance data).
Results: The results of the graduate assessment examination show that our incoming graduate students (7) came in even less prepared this year (shifting the overall average down by a whole percentage point over the past 6 years) than in the previous years. The exam score results also showed that our outgoing students (3) went out a bit less prepared this year (again the overall 6 year average shifted down by a whole percentage point). However, when we look at value added the overall averages went up from program entrance to exit by the same amount as in previous years. Particularly lowering the average of our exiting students this past year were two students who came in with virtually no background preparation whatsoever. The scores of the third exiting student (who was also a product of our own B.S. Physics program) were significantly higher than all of our 6 year averages and indicate that for those students who come in with appropriate background, that the goals of our program are being realized. In comparing the overall average entrance score with the overall average exit score, it can be seen that around a 37 percentage point increase took place throughout our program, and that in the area of classical mechanics was where the largest portion of that increase took place. This is also in qualitative agreement with earlier years' results.
The mid-program advising interviews indicated that students demonstrated competency in two or more areas of experimental physics by either their undergraduate physics laboratory supervision as part of their graduate assistantship assignments, their acceptable level of performance in an experimental physics course, or by their participation in a faculty-supervised experimental research project. Students produced either laboratory reports or conference presentations to document their accomplishments in this regard.
The graduate courses offered this year involved the use of Mathematica or a similar coprocessor in the solution of one or more homework assignments, and all of our students during their mid-program advising interviews documented an acceptable level of performance on one or more of these assignments. A number of graduate students also carried out conference presentations on their research projects where they clearly demonstrated their mastery of data processing, curve-fitting, and graphing procedures.
Feedback Loop: The changes that have been planned in reaction to the 2005-06 assessment data are as follows: primarily, we will continue our push to bring in a more highly qualified pool of applicants and candidates to our graduate program in physics. We will continue to require our students with weak backgrounds to make up their undergraduate deficiencies, and we have now set up plans to offer the second-semester courses in electromagnetism and quantum mechanics at the undergraduate level for our incoming graduate students, which due to faculty shortage could not be done in the past. This will commence during the fall semester 2006, along with a more advanced course in undergraduate mathematical physics that will also be required of our graduate students to improve their math skills. This will allow our students much better preparation prior to taking the graduate level courses, particularly in the two areas of electromagnetism and quantum mechanics, and these are the two theoretical core competency areas that obviously need the most background preparation and improvements as our assessment examination scores verified.
Students will have an understanding of:
- the political institutions and processes of the United States political system and of other major political systems;
- how nation states interact in the international political system;
- theories of politics;
And will develop:
- intellectual skills, including critical thinking and the ability to express cogently one’s ideas in writing.
Direct and Indirect Measurement: Students entering the graduate program in both the fall and the spring semesters take Part A (short answer) of the Comprehensive Exam.
Students nearing degree completion take the “official” Comprehensive Exam, consisting of Parts A (short answer) and B (long essay). Readers of the exams evaluate (using a five-point scale) student performance on the Assessment Exam (Part A, taken during the student’s first semester in the program) and compare it with the student’s performance on the Comprehensive Exam, taken at the end of the student’s tenure in the program.
Evaluation of thesis or two-paper option: assesses students’ ability to apply the concepts and theoretical frameworks of the discipline in oral (the formal defense) and written work (thesis or two-paper option).
Results: In reading Part A of the Assessment Exam for first-year graduate students and comparing them with graduate students in their last semester, there is very clear evidence of improved writing skills and an increase in substantive knowledge in the area of specialty. The differential, however, over the year and a half of graduate school is not as dramatic as when comparing undergraduates over a four-year time frame.
Feedback Loop: No feedback loop provided.
- Skill in evaluating the quality of research findings and in applying those findings to the clinical situation;
- Appreciation of professional issues including understanding of/adherence to professional ethics and standards;
- Understanding of the social, cultural and interpersonal factors that affect human behavior;
- Understanding of the intrapersonal factors (motivation, cognition, emotion) that affect human behavior;
- Skill in developing and using the therapeutic relationship; ability to therapeutically manage the ongoing interpersonal process including one’s own contribution to that process;
- Skill in gathering and integrating clinically relevant information using a variety of sources and procedures including clinical interviews and formal assessment instruments;
- Skill in developing clear, theoretically consistent formulations of client functioning, and deriving and implementing treatment plans based upon these formulations;
- Skill in assessing therapeutic progress based on formulations of client functioning;
- Skill in the appropriate use of a variety of therapeutic interventions;
- Understanding of the systemic characteristics of human systems and skill in intervening in such systems (e.g., families and therapy groups, communities);
- Skill in using professional supervision/consultation to address issues in ongoing treatment and to develop professional skills and understanding.
Direct and Indirect Measurement: The Clinical Skills Inventory (CSI) is completed on each student by all supervising faculty (meaning that a given student is typically rated by two or more faculty during any semester) for each of the three semesters that the students do clinical work in the department’s Psychology Clinic as a part of their practicum experience. This same scale is completed by the student’s primary internship supervisor twice during the internship--once near the midpoint and again at completion of the internship. Thus data from the CSI provides a view of the student’s level of achievement across all but one semester of their time in the program.
Other measures of student learning include the comprehensive examination, oral performance reviews including input from field supervisors twice during the internship, rate of placement into training appropriate jobs or doctoral programs, and faculty observations of student performance in various settings.
Results: The CSI data clearly show that students are demonstrating mastery of the skills involved in each of the 11 learning objectives as the minimum satisfactory rating during the practicum classes is 3.00 and during the internship it is 4.00. Moreover, the data suggest that students are showing steady progress in developing these skills across their time in the program. The only objective for which this is not true is “f,” the goal concerning skill in the use of formal assessment procedures which does not achieve a rating of 4.0 by the end of the internship.
Results from the comprehensive examination likewise show that students are demonstrating good mastery on all the learning objectives. Three students took the comprehensive examination during the study period, and all passed. One student was required to revise the case study because of problems with the case conceptualization and the understanding of the interpersonal process in the analysis of the transcript.
Field supervisors uniformly agreed that the students were well prepared to undertake the internship responsibilities, and the interns agreed with this perception.
The four students who completed the program during the study period are all employed full time in clinical service settings in positions appropriate to their training.
Feedback Loop: In response to these findings during the 2006-2007 academic year additional formal instruction on case conceptualization will be provided in staff meetings early in the fall semester, and students will be required, both in written case materials and in discussions during individual supervision, to track more carefully the connections between their conceptualization and understanding of interpersonal process and their clinical interventions, and to demonstrate consideration of relevant research in developing treatment plans and implementing interventions.
- Broad knowledge of the subfields of experimental psychology,
- Specialized knowledge of a particular subfield of experimental psychology,
- Knowledge of statistical techniques and research methods available for executing and interpreting research results,
- Skill in public speaking,
- Skill in written expression of ideas, and
- Ability to engage in original research.
Direct and Indirect Measurement: 1) First-year students’ scores on the “Department of Psychology Comprehensive Exam” (total score; sub-test score of student’s declared specialties in experimental psychology; and sub-test score on statistics/research methods section of comprehensive exam); 2) Scores from thesis advisors’ evaluations of the 6 learning objectives, collected upon students’ completion of master’s thesis; 3) Percentage of graduating students employed in psychology-related fields, among those who immediately entered the work force; and 4) Percentage of students continuing their educations, among those departing from the program in good standing
Results: Results indicate that students in the program met or exceeded generally accepted standards of performance. Their average scores on the department’s comprehensive exam routinely exceed the average score of undergraduate psychology seniors, who also take this exam. Their average scores in select content areas (broad knowledge of experimental psychology, specialized knowledge, and knowledge of statistics/research methods) tend to fluctuate around the B-range (3.00 on a 4.00-scale). Of particular importance is the exam score of 90% or better each year on the Statistics/Research Methods portion of the assessment exam. Virtually all graduates enter the workforce in a psychology-related field, or go on for advanced educational degrees.
Feedback Loop: Based on feedback from both students and faculty, new course offerings went into effect beginning in the fall 2005 semester. The assessment report submitted in Spring 2001 proposed two changes as a result of assessment findings. The first were changes in the advising system that have since been implemented. The effects of this change will continue to be monitored in subsequent years. The second change proposed in Spring 2001 as a result of assessment was modified promotion of the program to prospective students. That promotional process began during the 2001-2002 school year, with refreshed promotional materials (e.g., flyers and bulletin boards) and in contacts with prospective program applicants. Additionally, a web page for the program has been expanded and updated over the past two years. The effect and degree of success of these efforts will be monitored in subsequent years.
- Data-Based Decision Making and Accountability;
Consultation and Collaboration;
Socialization and Development of Life Skills;
Diversity in Development and Learning;
School and Systems Organization,
Policy Development, and Climate;
Prevention, Crisis Intervention, and Mental
Research and Program
School Psychology Practice and Development;
- Technology Standards
Direct and Indirect Measurement: Direct Measures: Students must complete all required coursework, practica, and internships; address all 11 standards in their portfolios; meet all requirements for certification as school psychologists; receive a final letter from their internship site supervisor indicating that they have completed all items on their internship plans; submit a signed copy of the completed internship plan; and submit a final copy of their research papers/thesis. Indirect Measures: alumni survey and alumni listserve.
Results: In 2005-2006, all students completed their coursework successfully. No one received more than nine hours of “C” and all students exceeded the 3.0 grade point average required.In 2005-2006, portfolios were completed by nine second year students and nine interns.
Of the nine interns: five did not have to redo any performance indicators, four were required to redo between one and five of the performance indicators (this is out of 90+ total indicators). Three of the four interns have completed the rewrites successfully at this time. Only two students were required to redo any of the knowledge indicators, one student was required to redo three knowledge indicators and the other student was required to redo two knowledge indicators (of more than 100 knowledge indicators total.)
Of the nine interns in 2005-2006, seven completed certification requirements for school psychology practice (five in Illinois, one in Iowa, and one in North Carolina). One intern did not complete the Illinois certification requirements as he is entering doctoral program. The remaining intern has not yet completed her research project, her internship, or her portfolio despite repeated efforts to encourage her to do so. All of the students from our program completing the Illinois State Certification Test have passed both parts of it.
Final letters from internship site supervisor as well as completed internship plans have been received for eight of the nine interns. As mentioned above, one intern has not yet completed her internship. Seven of the nine interns have completed final/approved copies of their research papers. One intern is currently completing the data analysis for his thesis. (Students have a research paper/thesis option in our program.) The ninth intern has not submitted a research paper to the faculty.
Feedback Loop: The program needs to undergo national accreditation in the fall. The most perplexing event this year with respect to the Assessment Plan has been the failure of one intern to complete her portfolio, her internship, and her research project. We do not have a policy in place for students who are performing adequately but simply do not finish some aspect of the program. The faculty will meet this summer at our annual retreat to decide what sort of policy might be needed to cover such cases.
Specifically, each student shall:
- possess knowledge of sociological theory, research methods, and statistics;
- have the ability to utilize sociological theory, methods, and statistics in written work;
- have a basic knowledge of program content;
- be able to state sociological problems;
- develop a research project;
- perform appropriate statistical analysis;
- synthesize qualitative and quantitative data;
- communicate effectively in written and oral forms.
Direct and Indirect Measurement: At the completion of the student's program, his/her thesis chair, or the primary reader of his/her non-thesis paper, completes a general assessment of the student's written work, including ordinal ranking of the student's knowledge of theory, methods and statistics as evidenced in his/her written work, i.e., thesis or non-thesis paper. At the completion of the student's program, his/her thesis chair, or the primary reader of his/her non-thesis paper, completes a general assessment of the student's oral presentation, including ordinal ranking of the student's knowledge of theory, methods and statistics as evidence in his/her oral defense of thesis or the student's presentation of his/her non-thesis option paper.
Results: The students met or exceeded expectations both in written and oral work on theses and non-thesis papers. In terms of assessment of both instruments A and B, 13 evaluators of completed master’s thesis and 11 evaluators of completed non-thesis papers reported no need for improvement on written work and oral presentation of these sociology master’s degree requirements. Notably, in terms of both written work and oral presentation, a majority of graduate sociology students who completed master’s theses were rated as exceeding expectations in terms of “conducting sociological analysis” and “elaborating theoretical significance” of their project. Concerning written and oral communications of non-thesis paper, a majority of graduate sociology students were rated as exceeding expectations.
In sum, assessment of graduate sociology students during the current evaluation period demonstrated that most graduate sociology students met or exceeded faculty expectations in forming research problems, conducting sociological analyses, and communicating in oral and written form. Notably, this same pattern was found in Graduate Sociology Assessment Reports during 1998-1999 and 2002-2003, 2003-2004, and 2004-2005.
Feedback Loop: Discussion among faculty at department meetings about the graduate sociology program in general during 2005-2006 has led to a consensus that the department desires to engage in a “Faculty Retreat” during 2006-2007. At this time, results of past graduate sociology assessment as well as consideration of undergraduate and graduate curriculum will be directly examined by faculty of the department in respect to studies of sociology and anthropology in the future.
There is no plan for assessment of the graduate program. It is sufficient to note here that the Post-Baccalaureate Graduate Certificate in Women’s Studies has enrolled nine students. The first course in the certificate program was offered in SP 06.