Comparar medias no es el medio: cuantificando el desempeño de los estudiantes en exámenes estandarizados
| Autor | Silvia C. Gómez Soler |
| Cargo | PhD Fellow, Maastricht University, The Netherlands |
| Páginas | 201-239 |
201
Cuad. admon.ser.organ. Bogotá (Colombia), 26 (46): 201-239, enero-junio de 2013
* The current paper results from the project Educación Superior en Colombia (Higher Education in Colombia), which,
conducted by the American University, started in January 2012 and ended in August of the same year. The opinions
expressed in this document correspond to the author and do not reect the opinión of Universidad Javeriana. Any
omission or mistake is the author’s responsibility. The paper was submitted for publication on 25-01-2013 and appro-
ved on 30-04-2013.
** PhD Fellow, Maastricht University, The Netherlands; Master of Arts in International Development (Development Eco-
nomics), American University, Washington, DC., U.S.A., 2012; Master’s degree in Economics, Ponticia Universi-
dad Javeriana, Bogota, Colombia, 2009; Bachelor’s degree in Economics, Ponticia Universidad Javeriana, Bogota,
Colombia, 2008. Instructor at the School of Economics, Ponticia Universidad Javeriana, Bogota, Colombia. E-mail:
sg8590a@american.edu; silvia.gomez@javeriana.edu.co
Comparing means Can be mean:
quantifying the performanCe
of students at standardized
exit exams*
Silvia C. Gómez Soler**
202 Cuad. admon.ser.organ. Bogotá (Colombia), 26 (46): 201-239, enero-junio de 2013
Silvia C. Gómez Soler
Comparing means can
be mean: quantifying the
performance of students
at standardized exit
exams
AbstrAct
The current work introduces an alternative analysis approach to standardized
exam results, which have been so far scrutinized by means of ranking systems
based on arithmetic mean comparisons. The Propensity Score Matching tech-
nique is applied to compare the results of equivalent students in the Colombian
eld-specic college exit exam (ECAES), which was introduced by the National
Education Ministry in 2003. The specic case of the students enrolled at Uni-
versidad Javeriana’s Business Administration Program is addressed, but the
methodology can be easily applied to other programs and universities. The re-
sults show a strong treatment effect of attending Universidad Javeriana on the
performance in this exam. In contrast to the results of previous ranking studies
based on simple average data, students from Universidad Javeriana were found
to perform better than equivalent students. This shows that the construction and
interpretation of those rankings might be awed.
Keywords: higher education, exit exams, economics of education, impact evaluation
JEL classication: I23, I25
Comparar medias no es
el medio: cuanticando
el desempeño de los
estudiantes en exámenes
estandarizados
resumen
El trabajo propone una metodología alternativa para analizar los resultados
de exámenes estandarizados, que hasta la fecha se han analizado mediante
rankings construidos a partir de promedios aritméticos. Utilizando la técnica
Propensity Score Matching se comparan los resultados de estudiantes con
características semejantes en los exámenes de Estado colombianos a nivel
universitario (ECAES), implementados por el Ministerio de Educación en 2003.
Se analiza el caso de los estudiantes de Administración de Empresas de la Uni-
versidad Javeriana, pero la metodología puede replicarse en otros programas y
universidades. Los resultados muestran un impacto signicativo del tratamiento
(ser estudiante de la Javeriana) sobre el desempeño en el ECAES. A diferencia
de los estudios que usan rankings a partir de promedios, los estudiantes de
la Javeriana tienen mejor desempeño que los estudiantes comparables, de-
mostrándose que la construcción e interpretación de rankings es inapropiada.
Palabras Claves: educación superior, exámenes de Estado, economía de la
educación, evaluación de impacto
Clasicación JEL: I23, I25
Comparar as médias não
é o meio: quanticando
o desempenho dos
estudantes em exames
padronizados
resumo
Neste trabalho, propõe-se uma metodologia alternativa para analisar os resul-
tados de exames padronizados que até o momento são analisados mediante
rankings construídos a partir de médias aritméticas. Ao utilizar a técnica Propen-
sity Score Matching, comparam-se os resultados de estudantes com caracte-
rísticas semelhantes nos exames de estado colombianos em nível universitário
(ECAES), implementados pelo Ministério da Educação em 2003. Analisa-se o
caso dos estudantes de Administração de Empresas da Universidad Javeriana,
mas a metodologia pode ser aplicada em outros programas e universidades.
Os resultados mostram um impacto signicativo do tratamento (ser estudante
da Javeriana) sobre o desempenho no ECAES. À diferença dos estudos que
usam rankings a partir de médias, os estudantes da Javeriana têm melhor des-
empenho que os estudantes comparáreis, o que demonstra que a construção
e a interpretação desses rankings não são apropriadas.
Palavras-chave: educação superior, exames de Estado, economia da educação,
avaliação de impacto.
Classicaҫão JEL: I23, I25
203
ComparinG meanS Can be mean: QuantifyinG the performanCe of StudentS at Standardized exit examS
Cuad. admon.ser.organ. Bogotá (Colombia), 26 (46): 201-239, enero-junio de 2013
Introduction
In 2003, as part of an initiative to improve
quality and accountability in Colombian
higher education, the National Education
Ministry introduced a eld-specic college
exit exam known as ECAES. The results of
the ECAES have been used in recent years
by the Colombian media to establish rank-
ings of what they consider to be the best aca-
demic programs. The purpose of this paper is
to develop an alternative analysis approach
to the one that, based on arithmetic mean
comparisons, is currently being used by the
media to quantify and rank the performance
of students in standardized exams. Although
the study focuses on the specic case of the
students enrolled at Universidad Javeriana’s
Business Administration Program, the meth-
odology can be easily applied to other uni-
versities and academic programs.
This paper examines and builds on the prob-
lems that might come up in attempting to
analyze the results of the ECAES exam us-
ing the rankings that have been constructed
by the Colombian media as a measure of ed-
ucational quality. It is argued that, although
the ECAES results are not a good proxy of
education quality, they are a good source of
information to compare student specic skills
at the national level (Popham, 1999). Com-
paring and ranking average results misses
the individual dimension of students’ per-
formance in this exam. In this respect, it has
been stated that “the construction of indices
by which institutions or departments are
ranked is arbitrary, inconsistent and based
on convenience measures” (Harvey, 2008).
Ranking according to average results is a
naïve approach because it assumes that the
change in outcome [ECAES score] is at-
tributable exclusively to the program being
evaluated. The main contribution of this pa-
per is to develop an alternative econometric
and analytical approach to the interpretation
of standardized exam results.
Propensity Score Matching is used to com-
pare the results of equivalent students and to
deal with methodological problems related to
self-selection bias. The results show a strong
treatment effect of attending Universidad Ja-
veriana on the performance in the ECAES
exam. The average treatment effect on the
treated varies from 5.863 to 9.051 points. If
we compare students that are similar in ob-
servable characteristics, Universidad Javeri-
ana students have signicantly higher scores
than other students in the ECAES exam,
which indicates that the ranking approach
might be misleading.
The robustness of the average treatment ef-
fect estimated in this paper is checked by
comparing the results from propensity score
matching with the results obtained using
other methodologies. Specically, the results
of least squares regressions and xed effects
regressions were used to test the consistency
of the propensity score matching results.
Due to the nature of the data analyzed in this
paper, it is particularly interesting to exam-
ine the heterogeneity of the impact. Given its
important policy implications, the specic
effect of learning at Universidad Javeriana
on certain variables like gender and socio-
economic stratum is particularly appealing.
Students from lower socioeconomic strata
Para continuar leyendo
Solicita tu prueba