Examination of Digital Literacy Levels of University Students
Keywords:
Digital Literacy, Literacy, University Students
Abstract
In this study, it was aimed to determine the digital literacy levels of university students. The study also aimed to examine the differentiation status of university students in terms of various variables of digital literacy levels. In addition, in the study, the digital literacy levels of university students were determined by gender, type of education (normal/secondary education), higher education level (associate / undergraduate/graduate), higher education program they are enrolled in, grade point average, class, average monthly income, social network with the most time spent. It is also aimed to examine the differentiation status in terms of the variables of average internet usage time during the day, the preferred tool for internet use, and the educational status of the parents. In this study, the survey model, which is one of the quantitative patterns, was used. The population of the study consisted of a total of 17,006 students, 11,500 undergraduate and 5,506 associate degree students, studying in 11 faculties, 4 colleges and 9 vocational colleges in the 2021-2022 academic year at a state university located in the Eastern Anatolia Region. The sample consisted of randomly selected 688 university students. The "Digital Literacy Scale" developed by Bayrakçı and Narmanlıoğlu in 2021 was used as a data collection tool. In the analysis process, nonparametric techniques were used since the data could not meet the assumptions of parametric techniques. As a result of the study, it was seen that the digital literacy levels of university students were low. In addition, it was concluded that there were statistically significant differences between university students' grade point averages, the social network they spend the most time on, the preferred tool in internet use, and the educational status of their parents and their digital literacy levels. Various suggestions were developed considering the results of the study.
References
Allen J. V. (2016). Developping digital literacy skills through guided reading istruction, The Florida Reading Journal, 51(1), 16–20.
Arık, K., & Kıyıcı, M. (2019). Lise öğrencilerinin sayısal okuryazarlık düzeylerinin belirlenmesi: Hendek Örneği. Sakarya University Journal of Education, 9(1), 47-68.
Arslan, S. (2019). İlkokullarda ve ortaokullarda görev yapan öğretmenlerin dijital okuryazarlık düzeylerinin çeşitli değişkenler açısından incelenmesi. (Unpublished Master Thesis). Sakarya University, Institute of Educational Sciences.
Aviram, R., & Alkalai, Y. (2006). Towards a theory of digital literacy: Three scenarios for the next steps. European Journal of Open Distance E-Learning, 9(1), 1-11.
Avrupa Yeterlilik Çerçevesi (AYÇ) Referans Seviyeleri. (2022). Access Address: http://www.myk.gov.tr/index.php/tr/ulusal-meslek-standartlar-ana/218-avrupayeterllkcerceves-ayc-referans-sevyeler. Date of Access: 08.05.2022.
Ateş, M., & Aşçı, A. U. (2021, Eylül). Okuryazarlık kavramı ve eğitimle ilişkili okuryazarlık türleri. VII. TURKCESS Uluslararası Eğitim ve Sosyal Bilimler Kongresi, 1-3 Eylül 2021. Access Address: https://www.researchgate.net/publication/356982618_ Okuryazarlik_Kavrami_ve_Egitimle_Iliskili_Okuryazarlik_Turleri. Date of Access: 09.05.2022.
Baltacı, A. (2018). Nitel araştırmalarda örnekleme yöntemleri ve örnek hacmi sorunsalı üzerine kavramsal bir inceleme. Bitlis Eren University Social Science Journal, 7(1), 231-274.
Bayrakçı, S., & Narmanlıoğlu, H. (2021). Digital literacy as whole of digital competences: Scale development study. Social Sceinces of Thought and Society, 4, 1-30.
Bingöl, H. (2022). Uzaktan eğitim sürecinde öğretmenlerin dijital okuryazarlık düzeyleri ile mesleki motivasyonlarının incelenmesi. (Unpublished Master Thesis). Zonguldak Bülent Ecevit Uiversity, Institute of Social Sciences.
Boyacı, Z. (2019). Öğretmen adaylarının yaşam boyu öğrenme eğilimleri ile dijital okuryazarlık düzeyleri arasındaki ilişki (Düzce Üniversitesi örneği). (Unpublished Master Thesis). Sakarya University, Institute of Educational Sciences.
Can, A. (2018). SPSS ile bilimsel araştırma sürecinde nicel veri analizi. Pegem Academy Publishing.
Castells, M. (2008). Enformasyon çağı: Ekonomi, toplum ve kültür, ağ toplumunun yükselişi. (Ebru Kılıç, Translator). Bilgi University Publishing.
Cote, T., & Milliner, B. (2018). A survey of FFL teachers’ digital literacy: A report from a Japanese Unıversity. Teaching English with Technology, 4, 71-89.
Çokluk, Ö., Şekercioğlu, G., & Büyüköztürk, Ş. (2018). Sosyal bilimler için çok değişkenli istatistik: SPSS ve LISREL uygulamaları. Pegem Academy Publishing.
Dinçer, S., Şenkal, O., & Sezgin, M. E.. (2013). Fatih projesi kapsamında öğretmen, öğrenci ve veli koordinasyonu ve bilgisayar okuryazarlık düzeyleri. Akademik Bilişim 2013 – XV. Akademik Bilişim Konferansı Bildirileri 23-25 Ocak 2013. Access Address: https://www.researchgate.net/publication/298192733_Fatih_Projesi_kapsaminda_ogretmen_ogrenci_ve_veli_koordinasyonu_ve_bilgisayar_okuryazarlik_duzeyleri. Date of Access: 08.05.2022.
Dinlemez, Ş. (2021). Türkçe öğretmeni adaylarının dijital okuryazarlık düzeyleri ile dijital vatandaşlık düzeyleri arasındaki ilişkinin belirlenmesi. (Unpublished Master Thesis). Çanakkale Onsekiz Mart University, School of Graduate Studies.
Doğan, D. (2022). Üniversite öğrencilerinin ve akademisyenlerin dijital okuryazarlık düzeylerinin incelenmesi. (Unpublished Master Thesis). Gazi University, Department of Educational Sciences.
Erkorkmaz, Ü., & Günay, O. (2002). Örnekleme yöntemleri ve bir uygulama. Erciyes University Journal of Health Sciences, 11(1), 36-44.
Eryılmaz, S., & Uluyol, E. (2015). Evaluation of FATİH project in the consideration of 21st century skills. Gazi University Journal of the Faculty of Education, 35(2), 209-229.
Eshet-Alkalai, Y., & Chajut, E. (2010). You can teach old dogs new tricks: the factors that affect changes over time in digital literacy. Journal of Information Technology Education, 9, 173–180.
European Commission (2017). DigComp 2.1: The digital competence framework for citizens with eight proficiency levels and examples of use. Luxembourg: Publications Office of the European Union.
European Union (2021). The Digital Education Action Plan 2021-2027. Retrieved from: https://ec.europa.eu/education/education-in-the-eu/digital-education-action-plan_en/ May 8 2022.
Fraenkel, J. R., & Wallen, N. E. (2006). How to design and evaluate research in education. (6th Edition). McGraw-Hill.
Goodfellow, R. (2011). Literacy, literacies, and the digital in higher education. Teaching in Higher Education, 16(1), 131-144.
Gül, G. (2007). Okuryazarlık sürecinde aile katılımının rolü. Journal of Special Education, 8(1), 17- 30.
Günay, G. (2022). Türkçe eğitiminde dijital okuryazarlık: Bir karma yöntem araştırması. (Unpublished Master Thesis). Çanakkale Onsekiz Mart University, School of Graduate Studies.
Hamutoğlu, N. B., Canan-Güngören, Ö., Kaya-Uyanık, G., & Gür-Erdoğan, D. (2017). Dijital okuryazarlık ölçeği: Türkçe ’ye uyarlama çalışması. Ege Journal of Education, 18(1), 408- 429.
Joint Information Systems Committee (JISC). (2014). Developping digital literacies. Retrieved from: https://www.jisc.ac.uk/guides/developing-digital-literacies. May 8 2022.
Juurakko-Paavola, T., Rontu, H., & Nelson, M. (2018, 12-14 November). Digital literacy and learning of languages and communicatıon in higher education-Case study from Finland. 11th annual International Conference of Education, Research and Innovation, 41-60.
Karabacak, Z. İ., & Sezgin, A. A. (2019). Türkiye’de dijital dönüşüm ve dijital okuryazarlık. Turkish Journal of Administration, 91(488), 319-343.
Karadeniz, E. Koşan, L., Geçgin, E., & Beyazgül, M. (2019). Üniversite Öğrencilerinin Finansal Okuryazarlık Düzeylerinin Analizi: Kafkas Üniversitesi Sarıkamış Kampüsünde Bir Araştırma. Celal Bayar University. Journal of Social Sciences, 17(2), 325-345.
Kerlinger, F. N., & Lee, H. B. (1999). Foundations of behavioral research. Harcourt College Publishers.
Kim, K. T. (2019). The structural relationship among digital literacy, learning strategies, and core competencies among South Korean college students. Educational Sciences: Theory And Practice,19(2), 9.
Kuru, E. (2019). Sosyal bilgiler öğretmen adaylarının dijital okuryazarlık kavramına ilişkin görüşleri. Turkish Studies, 14(3), 1629-1648. DOI: 10.29228/TurkishStudies.22563.
Özdamar, K. (2017). Eğitim, sağlık ve davranış bilimlerinde ölçek ve test geliştirme yapısal eşitlik modellemesi IBM SPSS, IBM SPSS AMOS ve MINTAB uygulamalı. Nisan Publishing.
Payton, S., & Hague, C. (2022). Digital literacy in practice. Case studies of primary and secondary classrooms. Futurelab innovation in education. Retrieved from: https://www.nfer.ac.uk/publications/FUTL06/FUTL06casestudies.pdf. 8. 05. 2022.
P21 Leadership States. (2022). Retrieved from: https://files.eric.ed.gov/fulltext/ED519462.pdf. May 8 2022.
Pehlivanlı, B. (2022). Dijital okuryazarlık bağlamında ortaokul Türkçe ders kitapları. (Unpublished Master Thesis). Kırıkkale University, Institute of Social Sciences.
Polizzi, G. (2020). Digital literacy and the national curriculum for England: Learning from how the experts engage with and evaluate online content. Computers & Education, 152, 1-13.
Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the fit of structural equation models: tests of significance and descriptive goodness of fit measures. Methods of PsychologicalResearch Online, 8(2), 23-74.
Tavşancıl, E. (2010). Tutumların ölçülmesi ve SPSS ile veri analizi. Nobel Publishing.
Taylor, A. R., & Dalal, H. (2017). Gender and information literacy: Evaluation of gender differences in a student survey of information sources. College & Research Libraries, 78(1), 90-113. DOI:10.5860/crl.78.1.90.
Thompson, K. (2016). Digital Literacy and the ICT Curriculum. Journal of Graduate Studies in Education, 8(1), 10-13.
Tomczyk, L. (2019). Digital literacy in the area of e-safety among teachers (Second stage of the primary school) in Poland. Conference proceedings of e-Learning and Software for Education (eLSE), 15,130-135.
Türkiye Yeterlikler Çerçevesi, (2017). Access Address: https://www.myk.gov.tr/index.php/tr/turkiye-yeterlilikler-cercevesi. Date of Access: 08.05.2022.
Vélez, A. P., Olivencia, J. J. L. & Zuazua, I. I. (June, 2017). The role of adults in children digital literacy. Procedia-Social and Behavioural Sciences, 7th International Conference on Intercultural Education “Education, Health and ICT for a Transcultural World”, EDUHEM 2016, 15-17 June 2016. Almeria, Spain 887-892. Retrieved from: https://core.ac.uk/download/pdf/157753984.pdf. 08. 05.2022.
Yaman, C. (2019). Sosyal bilgiler öğretmen adaylarının dijital okuryazarlık düzeylerinin incelenmesi (Niğde Ömer Halisdemir Üniversitesi Örneği). (Unpublished Master Thesis). Niğde Ömer Halisdemir University, Department of Turkish and Social Sciences Education.
Yıldırım, F., & Naktiyok, S. (2017). The mediating role of organizational support in the effect of transformational leadership on employee empowerment. Polish Journal of Management Studies, 16(1): 292-303.
Yüce, A., & Korucuk, N. (2020). Coğrafi işaretli ürünleri satın almaya yönelik tutum ölçeği; Geçerlik ve güvenirlik çalışması. ISARDER, 12(4), 3569-3581. DOI: https://doi.org/10.20491/isarder.2020.1059.
Zher, S. Y. (2017). Developping a digital literacy scale & measuring digital divide using PIAAC data. 4 th PIAAC International Conference Singapore 21- 23 November 2017. Retrieved from: https://www.ial.edu.sg/content/dam/projects/tms/ial/Access-research/piaac/YeeZherSheng_SanChyeCheng_Developing%20a%20digital%20literacy%20scale%20and%20measuring%20digital%20divide%20using%20PIAAC%20data.pdf.08. 05. 2022
Arık, K., & Kıyıcı, M. (2019). Lise öğrencilerinin sayısal okuryazarlık düzeylerinin belirlenmesi: Hendek Örneği. Sakarya University Journal of Education, 9(1), 47-68.
Arslan, S. (2019). İlkokullarda ve ortaokullarda görev yapan öğretmenlerin dijital okuryazarlık düzeylerinin çeşitli değişkenler açısından incelenmesi. (Unpublished Master Thesis). Sakarya University, Institute of Educational Sciences.
Aviram, R., & Alkalai, Y. (2006). Towards a theory of digital literacy: Three scenarios for the next steps. European Journal of Open Distance E-Learning, 9(1), 1-11.
Avrupa Yeterlilik Çerçevesi (AYÇ) Referans Seviyeleri. (2022). Access Address: http://www.myk.gov.tr/index.php/tr/ulusal-meslek-standartlar-ana/218-avrupayeterllkcerceves-ayc-referans-sevyeler. Date of Access: 08.05.2022.
Ateş, M., & Aşçı, A. U. (2021, Eylül). Okuryazarlık kavramı ve eğitimle ilişkili okuryazarlık türleri. VII. TURKCESS Uluslararası Eğitim ve Sosyal Bilimler Kongresi, 1-3 Eylül 2021. Access Address: https://www.researchgate.net/publication/356982618_ Okuryazarlik_Kavrami_ve_Egitimle_Iliskili_Okuryazarlik_Turleri. Date of Access: 09.05.2022.
Baltacı, A. (2018). Nitel araştırmalarda örnekleme yöntemleri ve örnek hacmi sorunsalı üzerine kavramsal bir inceleme. Bitlis Eren University Social Science Journal, 7(1), 231-274.
Bayrakçı, S., & Narmanlıoğlu, H. (2021). Digital literacy as whole of digital competences: Scale development study. Social Sceinces of Thought and Society, 4, 1-30.
Bingöl, H. (2022). Uzaktan eğitim sürecinde öğretmenlerin dijital okuryazarlık düzeyleri ile mesleki motivasyonlarının incelenmesi. (Unpublished Master Thesis). Zonguldak Bülent Ecevit Uiversity, Institute of Social Sciences.
Boyacı, Z. (2019). Öğretmen adaylarının yaşam boyu öğrenme eğilimleri ile dijital okuryazarlık düzeyleri arasındaki ilişki (Düzce Üniversitesi örneği). (Unpublished Master Thesis). Sakarya University, Institute of Educational Sciences.
Can, A. (2018). SPSS ile bilimsel araştırma sürecinde nicel veri analizi. Pegem Academy Publishing.
Castells, M. (2008). Enformasyon çağı: Ekonomi, toplum ve kültür, ağ toplumunun yükselişi. (Ebru Kılıç, Translator). Bilgi University Publishing.
Cote, T., & Milliner, B. (2018). A survey of FFL teachers’ digital literacy: A report from a Japanese Unıversity. Teaching English with Technology, 4, 71-89.
Çokluk, Ö., Şekercioğlu, G., & Büyüköztürk, Ş. (2018). Sosyal bilimler için çok değişkenli istatistik: SPSS ve LISREL uygulamaları. Pegem Academy Publishing.
Dinçer, S., Şenkal, O., & Sezgin, M. E.. (2013). Fatih projesi kapsamında öğretmen, öğrenci ve veli koordinasyonu ve bilgisayar okuryazarlık düzeyleri. Akademik Bilişim 2013 – XV. Akademik Bilişim Konferansı Bildirileri 23-25 Ocak 2013. Access Address: https://www.researchgate.net/publication/298192733_Fatih_Projesi_kapsaminda_ogretmen_ogrenci_ve_veli_koordinasyonu_ve_bilgisayar_okuryazarlik_duzeyleri. Date of Access: 08.05.2022.
Dinlemez, Ş. (2021). Türkçe öğretmeni adaylarının dijital okuryazarlık düzeyleri ile dijital vatandaşlık düzeyleri arasındaki ilişkinin belirlenmesi. (Unpublished Master Thesis). Çanakkale Onsekiz Mart University, School of Graduate Studies.
Doğan, D. (2022). Üniversite öğrencilerinin ve akademisyenlerin dijital okuryazarlık düzeylerinin incelenmesi. (Unpublished Master Thesis). Gazi University, Department of Educational Sciences.
Erkorkmaz, Ü., & Günay, O. (2002). Örnekleme yöntemleri ve bir uygulama. Erciyes University Journal of Health Sciences, 11(1), 36-44.
Eryılmaz, S., & Uluyol, E. (2015). Evaluation of FATİH project in the consideration of 21st century skills. Gazi University Journal of the Faculty of Education, 35(2), 209-229.
Eshet-Alkalai, Y., & Chajut, E. (2010). You can teach old dogs new tricks: the factors that affect changes over time in digital literacy. Journal of Information Technology Education, 9, 173–180.
European Commission (2017). DigComp 2.1: The digital competence framework for citizens with eight proficiency levels and examples of use. Luxembourg: Publications Office of the European Union.
European Union (2021). The Digital Education Action Plan 2021-2027. Retrieved from: https://ec.europa.eu/education/education-in-the-eu/digital-education-action-plan_en/ May 8 2022.
Fraenkel, J. R., & Wallen, N. E. (2006). How to design and evaluate research in education. (6th Edition). McGraw-Hill.
Goodfellow, R. (2011). Literacy, literacies, and the digital in higher education. Teaching in Higher Education, 16(1), 131-144.
Gül, G. (2007). Okuryazarlık sürecinde aile katılımının rolü. Journal of Special Education, 8(1), 17- 30.
Günay, G. (2022). Türkçe eğitiminde dijital okuryazarlık: Bir karma yöntem araştırması. (Unpublished Master Thesis). Çanakkale Onsekiz Mart University, School of Graduate Studies.
Hamutoğlu, N. B., Canan-Güngören, Ö., Kaya-Uyanık, G., & Gür-Erdoğan, D. (2017). Dijital okuryazarlık ölçeği: Türkçe ’ye uyarlama çalışması. Ege Journal of Education, 18(1), 408- 429.
Joint Information Systems Committee (JISC). (2014). Developping digital literacies. Retrieved from: https://www.jisc.ac.uk/guides/developing-digital-literacies. May 8 2022.
Juurakko-Paavola, T., Rontu, H., & Nelson, M. (2018, 12-14 November). Digital literacy and learning of languages and communicatıon in higher education-Case study from Finland. 11th annual International Conference of Education, Research and Innovation, 41-60.
Karabacak, Z. İ., & Sezgin, A. A. (2019). Türkiye’de dijital dönüşüm ve dijital okuryazarlık. Turkish Journal of Administration, 91(488), 319-343.
Karadeniz, E. Koşan, L., Geçgin, E., & Beyazgül, M. (2019). Üniversite Öğrencilerinin Finansal Okuryazarlık Düzeylerinin Analizi: Kafkas Üniversitesi Sarıkamış Kampüsünde Bir Araştırma. Celal Bayar University. Journal of Social Sciences, 17(2), 325-345.
Kerlinger, F. N., & Lee, H. B. (1999). Foundations of behavioral research. Harcourt College Publishers.
Kim, K. T. (2019). The structural relationship among digital literacy, learning strategies, and core competencies among South Korean college students. Educational Sciences: Theory And Practice,19(2), 9.
Kuru, E. (2019). Sosyal bilgiler öğretmen adaylarının dijital okuryazarlık kavramına ilişkin görüşleri. Turkish Studies, 14(3), 1629-1648. DOI: 10.29228/TurkishStudies.22563.
Özdamar, K. (2017). Eğitim, sağlık ve davranış bilimlerinde ölçek ve test geliştirme yapısal eşitlik modellemesi IBM SPSS, IBM SPSS AMOS ve MINTAB uygulamalı. Nisan Publishing.
Payton, S., & Hague, C. (2022). Digital literacy in practice. Case studies of primary and secondary classrooms. Futurelab innovation in education. Retrieved from: https://www.nfer.ac.uk/publications/FUTL06/FUTL06casestudies.pdf. 8. 05. 2022.
P21 Leadership States. (2022). Retrieved from: https://files.eric.ed.gov/fulltext/ED519462.pdf. May 8 2022.
Pehlivanlı, B. (2022). Dijital okuryazarlık bağlamında ortaokul Türkçe ders kitapları. (Unpublished Master Thesis). Kırıkkale University, Institute of Social Sciences.
Polizzi, G. (2020). Digital literacy and the national curriculum for England: Learning from how the experts engage with and evaluate online content. Computers & Education, 152, 1-13.
Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the fit of structural equation models: tests of significance and descriptive goodness of fit measures. Methods of PsychologicalResearch Online, 8(2), 23-74.
Tavşancıl, E. (2010). Tutumların ölçülmesi ve SPSS ile veri analizi. Nobel Publishing.
Taylor, A. R., & Dalal, H. (2017). Gender and information literacy: Evaluation of gender differences in a student survey of information sources. College & Research Libraries, 78(1), 90-113. DOI:10.5860/crl.78.1.90.
Thompson, K. (2016). Digital Literacy and the ICT Curriculum. Journal of Graduate Studies in Education, 8(1), 10-13.
Tomczyk, L. (2019). Digital literacy in the area of e-safety among teachers (Second stage of the primary school) in Poland. Conference proceedings of e-Learning and Software for Education (eLSE), 15,130-135.
Türkiye Yeterlikler Çerçevesi, (2017). Access Address: https://www.myk.gov.tr/index.php/tr/turkiye-yeterlilikler-cercevesi. Date of Access: 08.05.2022.
Vélez, A. P., Olivencia, J. J. L. & Zuazua, I. I. (June, 2017). The role of adults in children digital literacy. Procedia-Social and Behavioural Sciences, 7th International Conference on Intercultural Education “Education, Health and ICT for a Transcultural World”, EDUHEM 2016, 15-17 June 2016. Almeria, Spain 887-892. Retrieved from: https://core.ac.uk/download/pdf/157753984.pdf. 08. 05.2022.
Yaman, C. (2019). Sosyal bilgiler öğretmen adaylarının dijital okuryazarlık düzeylerinin incelenmesi (Niğde Ömer Halisdemir Üniversitesi Örneği). (Unpublished Master Thesis). Niğde Ömer Halisdemir University, Department of Turkish and Social Sciences Education.
Yıldırım, F., & Naktiyok, S. (2017). The mediating role of organizational support in the effect of transformational leadership on employee empowerment. Polish Journal of Management Studies, 16(1): 292-303.
Yüce, A., & Korucuk, N. (2020). Coğrafi işaretli ürünleri satın almaya yönelik tutum ölçeği; Geçerlik ve güvenirlik çalışması. ISARDER, 12(4), 3569-3581. DOI: https://doi.org/10.20491/isarder.2020.1059.
Zher, S. Y. (2017). Developping a digital literacy scale & measuring digital divide using PIAAC data. 4 th PIAAC International Conference Singapore 21- 23 November 2017. Retrieved from: https://www.ial.edu.sg/content/dam/projects/tms/ial/Access-research/piaac/YeeZherSheng_SanChyeCheng_Developing%20a%20digital%20literacy%20scale%20and%20measuring%20digital%20divide%20using%20PIAAC%20data.pdf.08. 05. 2022
Published
2022-06-24
How to Cite
Kaya, H., & Korucuk, M. (2022). Examination of Digital Literacy Levels of University Students. Dinamika Ilmu, 22(1), 167-186. https://doi.org/10.21093/di.v22i1.4523
Section
Articles
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication, with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work
