Construct validation of a general English language needs analysis instrument

Shiken: JALT Testing & Evaluation SIG Newsletter
Vol. 10 No. 2. Dec. 2006. (p. 2 - 11) [ISSN 1881-5537]

Construct validation of a
general English language needs analysis instrument

KUMAZAWA Takaaki
Kanto Gakuin University

Introduction

Literature Review

Broadly defined, needs analysis (NA) is a procedure to collect information about learners' needs (Richards, 2001). The importance of NA is emphasized in English for Specific Purpose (Hutchinson & Waters, 1987) and English for Academic Purposes (Jordan, 1997), and also in general language courses espousing learner-centered curricula (Nunan, 1988; Tudor, 1996), task-based curricula (Long & Crookes, 1992), as well as performance-assessment (Norris, Brown, Hudson, & Yoshioka, 1988).
NA is considered a crucial component of systematic curriculum development. In Brown's (1995, p. 21) systematic curriculum development model it is the first phase of an ongoing quality control process (see Table 1). Brown (1995, p. 21) defines NA as

the systematic collection and analysis of all relevant information necessary to satisfy the language learning requirements of the students within the context of the particular institutions involved in the learning situation.

In the initial NA phase, administrators collect and analyze information about students' needs in order to design sound, defensible objectives-which is the second phase of Brown's five phase model depicted in Figure 1. That is, based on this model, the purpose of conducting NA is to systematically gather information in order to design objectives. While goals are "general statements about what must be accomplished in order to attain and satisfy students' needs," objectives refer to "precise statements about what content or skills the students must master in order to attain a particular goal" (Brown, 1995, p. 21). Thus, objectives have to be derived from corresponding goals. Therefore, it is essential for a language program to have well-defined goals so that the subsequent evaluation instruments can accurately measure the extent to which students have mastered the goals. Administrators can select the goals that students feel the need to learn and extrapolate these in terms of specific objectives which represent a concrete manifestation of those goals. NA is generally administered to a particular target group of students at a program-level. For the administration to a large number of students, a questionnaire is the most frequently used and efficient method to elicit responses.

[ p. 2 ]

Figure 1. The systematic curriculum development model proposed by Brown (1995)

A number of articles have been published on NA such as those by Basturkmen (1998), Berwick (1989), and West (1994). However, the actual detailed studies on this topic are scarce (e. g., Iwai, Kondo, Lim, Ray, Shimizu, & Brown, 1999; Chaudron, et al., 2005). In Japanese contexts, quiet a few studies have explored variables such as students' bio data, motivation, strategies, learning beliefs, learning styles and preference, and perceived difficulty in learning (Hiromori, 2003; Kikuchi, 2005; Kuwabara, Nakanishi, & Komai, 2005; Robson & Midorikawa, 2001; and Suzuki & Kumazawa, 2006). These instruments were employed to investigate individual differences among student respondents. Especially, teachers can make use of such information to better discern characteristics of their students and subsequently making lessons more satisfying for them by addressing their needs. For instance, if students prefer working in pairs to small groups, teachers can provide more pair-work activities. However, it is often difficult to translate subjective student preferences into course objectives. One instrument that reputedly does this can be found in Busch, Elsea, Gruba, and Johnson (1992). In one of the sections called "present student needs", they list nine items in which the expression "need" was included as part of item description wordings so that respondents could specify the extent of their needs with concision (Busch et. al., 1992, p. 18). Moreover, in 2004 Kusangi and Kumazawa made an attempt to develop and validate an NA instrument with these features. In their study, a Rasch analysis was conducted assuming that all 75 items were unidimensional. The results indicated that several items were misfitting, and the instrument lacked validity. One of the confounding factors was that many terms had a variety of ambiguous wordings which likely tapped into a number of constructs. Ideally, precise wording which taps into a single construct should be used.

Research Questions

Since the 2004 version of the NA instrument designed by Kusanagi and Kumazawa was not valid, this study sought to develop a revised version of a NA instrument which overcame some of the limitations of the previous instrument. This version of the NA instrument, called the General English Language Needs Analysis Instrument (GELNA), will be examined in this paper. The specific research question is:

To what extent is Version 1 of the GELNA valid based on confirmatory factor analyses?

Methods

[ p. 3 ]

The General English Language Needs Analysis Instrument (GELNA, Ver. 1)

Most of the EFL programs for university students have an avowed common goal to improve students' English proficiency. According to the systematic curriculum model (Brown, 1995), the objectives should focus on achieving this goal. This particular instrument was developed in 2005 in two general English programs for university students with the following courses: speaking, listening, reading, writing, culture-oriented, test-preparation, and computer-assisted language learning (CALL). The GELNA has seven sections which correspond to the seven courses offered in the programs. The items in the GELNA represent the avowed goals of the course (see Table 1). That is, this instrument was designed to measure the extent to which the overall curricular goals matched the students' perceptions of their own learning needs. The primary purpose of this instrument was not to obtain information on the students' bio data, motivation, strategies, and learning styles, but to see how congruent the curricular goals were with the avowed student needs. For each of the sections of the GELNA, two or three goals were included in a 6-point Likert scale. Version 1 of this instrument consisted of 20 items, as shown in Table 1.

Table 1. English translation of the items in the GELNA, Ver. 1

	Items
	Section 1: Culture-oriented Course
1.	I need to learn concepts in cross-cultural communication such as cultural values.
4.	I need to practice many activities that make me understand my own culture and aware of cultural differences.
13.	I need to learn how to handle situations when I encounter cross-cultural differences.
	Section 2: CALL Course
10.	I need to practice making my homepage in English.
17.	I need to take a class that uses authentic audio-visual materials such as videos, CDs, and audio*.
18.	I need to take a class that uses computers for learning.
	Section 3: Listening Course
2.	I need to practice listening to be able to understand stress pattern and intonation.
5.	I need to practice watching dramas in English in order to be able to understand the content.
15.	I need to practice listening extensively to get the main ideas.
	Section 4: Reading Course
3.	I need to learn reading skills such as reading rapidly and getting the gist.
6.	I need to practice reading by focusing on the grammar of English texts and translating them into Japanese.
7.	I need to study the structures of English sentences.
	Section 5: Speaking Course
8.	I need to learn to discuss issues effectively in English.
12.	I need to practice making a speech and presenting ideas in English.
16.	I need to take a class in which my final grading is decided based on my score on test performance such as a speech.
	Section 6: Test-preparation Course
19.	I need to take a class where I solve many TOEIC, TOEFL, and STEP questions.
20.	I need to learn test-taking strategies to solve problems in TOEIC, TOEFL, and STEP.
	Section 7: Writing Course
9.	I need to practice writing papers in English.
11.	I need to practice writing business letters in English.
14.	I need to take a class in which my final grading is decided based on the result of my paper.

[ p. 4 ]

The differences between the GELNA and the previous NA instruments discussed earlier in this paper was that the GELNA clearly stated the avowed goals of the language programs and the items were designed to specifically tap into theoretical constructs by embedding the phrase "need to" in the wording. For instance, the wording for Item 1 could be translated as "I need to learn concepts in cross-cultural communication such as cultural values." Learning concepts in cross-cultural communication was a clearly stated goal in both language programs where this instrument was used. Since needs, wants, and values likely pertain different constructs, by specifically stating 'need to' in each respective item statement, the GELNA attempt to avoid some of the ambiguity of earlier NA instruments.

Participants

The GELNA instrument was administered at two private universities in Japan in July 2005. In one mid-ranked university, 155 first-, second-, and third-year students in the English department took the test. In the general English program of that school, four kinds of courses are offered: reading, communicative grammar, computer writing skills, and test-preparation courses. The curriculum is not systematically unified. In the other high-ranked university, 32 first-year students in the economics department and 43 first-year students in the tourism department participated in this study. This proficiency-based general English program that offers communication-oriented and culture-oriented courses is systematically well-unified, which means assigned objectives and textbooks are set for all the classes. In total, 230 participants participated in this study.

Procedures

Version 1 of the GELNA was distributed to the participants in class at the end of the first semester in 2005. The distributors instructed students how to fill out the instrument, explained that the results had no connection to their grades, and students were told that they could leave if they did not want to complete the questionnaire. However, all the students were cooperative and remained in their seats to fill out their responses. Twenty minutes to answer the instrument were allocated for them.

[ p. 5 ]

Analysis

As an initial step, descriptive statistics were examined and the necessary data screening was carried out as described in Molloy and Newfields (2005, p. 3). Eleven items were found to be skewed, but were not far apart from the acceptable range. Thus, the violation of normality did not seem to be problematic. Ten cases were excluded because they had missing values. Fourteen cases were found to be outliers and excluded. Two cases were also eliminated because the participants circled the same response for all the 20 items, suggesting that they did not complete the instrument seriously. The total number of cases used for this analysis was 204 (n = 204). The reliability for the GELNA was checked using Cronbach alpha. For the validation, a confirmatory factor analysis was carried out because the GELNA hypothetically had seven sections. Maximum likelihood was used as the parameter estimation technique. The p-value was set at .05. SPSS and AMOS were used for the analyses.

Results

Table 2 shows the descriptive statistics. The mean scores suggest that the participants felt that CALL and writing courses are least needed. Conversely, culture-oriented, listening, and test-preparation courses were considered the most needed. Notice that the mean for Item 2 was the highest. The participants strongly felt that they needed to practice listening to be able to understand the stress and intonation accurately. The least needed goal was to learn to make a homepage in English.
Factor patterns are displayed in Table 2 as well as in Figure 2. Figure 2 shows the result of the confirmatory factor analysis model. Squares indicate observable variables which are, in this case, the items in the GELNA. Ovals represent latent variables or factors that are theoretical constructs. The single-headed arrows from the ovals and squares are called paths. The path coefficients signify the degree of causality between observable and latent variables. In a confirmatory factor analysis model, these path coefficients can be interpreted as factor patterns that are like factor loadings. In exploratory factor analysis they show the relationship between observable and latent variables. The double-headed arrows between two latent variables are like correlation coefficients. The small circles attached to observable variables are errors inherent in the items. The factor patterns are moderate or high, ranging from .41 to .99. Notice that Item 5 was excluded in Figure 2 because the factor pattern was lower than .30 in the initial run for a confirmatory factor analysis. The fit indices indicate the degree of adequacy of a model to the data. The χ² was significant, and the GFI, AGFI, CFI, and RMSEA were .89, .84, .92, and .07, respectively. These results showed that the model was fairly adequate.
The sectional Cronbach alpha internal consistency coefficients were moderate or high, ranging from .44 to .96. The number of items within each section was small but it was still reliable except for Section 3. The coefficient for the 19-item GELNA was high at .87. In addition, the coefficient for the data set without the data screening (n = 231) was also .87.

Table 2. Descriptive statistics for Version 1 of the GELNA (n = 204)

Item #	Items	M	SD	Factor Pattern
Section 1: Culture-oriented Course (α = .76)
1.	I need to learn concepts in cross-cultural communication such as cultural values.	4.69	0.95	.74
4.	I need to practice many activities that make me understand my own culture and aware of cultural differences.	4.42	1.08	.77
13.	I need to learn how to handle situations when I encounter cross-cultural differences.	4.56	1.01	.65
Section 2: CALL Course (α = .63)
10.	I need learn how to make a web page in English.	3.07	1.24	.65
17.	I need to take a class that uses authentic audio-visual materials such as videos, CDs, and audio.	4.87	1.02	.47
18.	I need to take a class that uses computers for learning.	3.96	1.14	.71
Section 3: Listening Course (α = .44)
2.	I need to practice listening to be able to understand stress pattern and intonation.	5.08	0.90	.41
5.	I need to practice watching dramas in English in order to be able to understand the content.	4.69	1.04	*
15.	I need to practice listening extensively to get the main ideas.	4.75	0.98	.70
Section 4: Reading Course (α = .65)
3.	I need to learn reading skills such as reading rapidly and getting the gist.	4.64	0.90	.48
6.	I need to practice reading by focusing on the grammar of English texts and translating them into Japanese.	3.79	1.12	.80
7.	I need to study the structures of English sentences.	3.95	1.00	.61
Section 5: Speaking Course (α = .80)
8.	I need to learn to discuss issues effectively in English.	4.33	1.20	.76
12.	I need to practice making a speech and presenting ideas in English.	4.42	1.20	.80
16.	I need to take a class in which my final grading is decided based on my score on test performance such as a speech.	4.14	1.07	.71
Section 6: Test-preparation course (α = .96)
19.	I need to take a class where I solve many TOEIC, TOEFL, and STEP questions.	4.81	1.16	.93
20.	I need to learn test-taking strategies to solve problems in TOEIC, TOEFL, and STEP.	4.81	1.19	.99
Section 7: Writing Course (α = .64)
9.	I need to practice writing papers in English.	4.12	1.17	.77
11.	I need to practice writing business letters in English.	3.71	1.29	.67
14.	I need to take a class in which my final grading is decided based on the result of my paper.	4.11	1.05	.41

Note: *Item 5 was excluded because the factor pattern was below .30 in the initial run for a confirmatory factor analysis.
The reliability for the remaining 19 items was .87. The items are based on a 6-point Likert scale.

[ p. 6 ]

Figure 2. A confirmatory factor analysis model for Version 1 of the GELNA

Discussion

The purpose of this study was to examine the construct validity of the first version of the GELNA. As a result of these analyses, it was found that except one section, the rest of the instrument had moderate to high reliability coefficients. The coefficient for the entire instrument was also high. The factor patterns were moderate to high. Also, the confirmatory factory analysis model showed an adequate fit with the data. Taken together, these results seem to justify the construct validity of the GELNA.
Although there is statistical support for the construct validity of the GELNA, it is also important that content validity issues be addressed. In the process of this evaluation, four possible adjustments on the GELNA are discussed: (a) negative questions, (b) target culture, (c) specific statements, and (d) double-barreled questions. In general, negative or double-negative questions should be avoided because they can lead to misinterpretation or confusion (Brown, 2001, p. 47). However, it might be reasonable to include negative questions. In the GELNA, almost all of the questions were worded in such a way to make it sound like all the goals were needed for students' learning. This might lead to an expectancy effect resulting in unnaturally high scores, so including some statements counter-suggestive of the goals might reduce any expectancy effect (Dörnyei, 2003). In this study, the data from two respondents were excluded because they circled the same response throughout the survey. To detect students who might be randomly circling responses, negative questions can be used to see if their reverse coded responses to negatives questions are the same as their responses to the corresponding positive questions.
In Items 1, 4, and 13 about cross-cultural difference, the target countries were broadly set as foreign countries (Jp. takoku), so the target countries were needed to be specified. When many young Japanese hear the words such as takoku or kokusai they think of the North America and Western Europe, but there are actually over 192 countries in the world: many of outside of those regions. Japanese might be willing to learn about the cultures of some countries, but not others. This can be easily fixed by replacing takoku to terms such as North America or a specific country.
Goals are general statements of the program's purpose, and the items in the GELNA should not be too specific. However, there are some minor ambiguities in the instrument which need to be addressed. For example, Item 2 did not specify which language the statement pertained to. Japanese university students might infer that it was about English, but the matter should be more clearly stated.

[ p. 7 ]

Moreover, Item 7 about learning structure of English sentences is general. Once again Japanese university students might infer that it was about learning grammatical structure of English sentences. However, it could mean learning the organizational structure of English essays.
Double-barreled questions ask two or more questions simultaneously (Brown, 2001, p. 49) and should be avoided to interpret accurately. For instance, Item 19 of Version 1 of the GELNA asked if the respondents felt they needed to take a class focusing on the TOEIC, TOEFL, and STEP. This item essentially asks three questions at the same time. Some respondents might feel they need to take TOEFL, but not the STEP. Such questions tend to confuse the respondents and can confound the results.

Conclusion

In developing a general language curriculum systematically, needs analysis instruments such as the GELNA may be useful especially in developing sound, defensible goals. Some institutions might prefer to tailor the GELNA to suit their particular goals. Although the GELNA is certainly not the only needs analysis tool, the fact that it was developed in a Japanese university setting might make it particularly useful to English educators in Japanese universities.
Based on the discussions about the content validity issues, a revised form of the GELNA has been developed and appears in Appendix 2. This version differs from the original version in the following ways: (1), including negative coded questions (2), avoiding double-barreled questions, (3) making more specific statements.
The areas for further research include the following questions: (1) How do students' responses to the revised GELNA differ from those to the original version?, and (2) To what extent can the reliability and construct validity of the revised GELNA be improved?
It is hoped that these questions be examined further to overcome the limitations of the GELNA.

Acknowledgement

Special thanks to Professor JD Brown for kindly offering advice on the scope of this article.

References

Basturkmen, H. (1998, October-December). A needs analysis project at Kuwait University. English Teaching Forum, 36 (4) 2. Retrieved on September 10, 2006 from http://exchanges.state.gov/forum/vols/vol36/no4/p2.htm

Berwick, R. (1989). Needs assessment in language programming: From theory to practice. In R. K. Johnson (Ed.), The second language curriculum (pp. 48-62). Cambridge: Cambridge University Press.

Brown, J. D. (1995). The elements of language curriculum: A systematic approach to program development. Boston, MA: Heinle & Heinle.

Brown, J. D. (2001). Using surveys in language programs. Cambridge: Cambridge University Press.

Busch, M., Elsea, M., Gruba, P., & Johnson, F. (1992). A study of the needs, preferences, and attitudes concerning the learning and teaching English proficiency as expressed by students and teachers at Kanda University. The Journal of Kanda University of International Studies, 6, 174-235.

Chaudron, C., et al. (2005). A task-based needs analysis of a tertiary Korean as a foreign language program. In M. Long (Ed.), Second Language Needs Analysis (pp. 225-261). Cambridge: Cambridge University Press.

Dörnye, Z. (2003). Questionnaires in second language research: Construction, administration, and processing. Mahwah, NJ: Lawrence Erlbaum Associates.

[ p. 8 ]

Hiromori, T. (2003). What enhances language learners' motivation?: High school English learners' motivation from the perspective of self-determination theory. JALT Journal, 25 (2), 173-186.

Hutchinson, T., & Waters, A. (1987). English for specific purpose: A learning-centered approach. Cambridge: Cambridge University Press.

Iwai, T., Kondo, K., Lim, D.S.J., Ray, G.E., Shimizu, H., & Brown, J.D. (1999). Japanese language needs assessment 1998-1999 (NFLRC Network #13). Honolulu: University of Hawai'i, Second Language Teaching & Curriculum Center. Retrieved on August 12, 2006 from http://www.nflrc.hawaii.edu/NetWorks/NW13/

Jordan, R. (1997). English for academic purposes: A guide and resource book for teachers. Cambridge: Cambridge University Press.

Kikuchi, K. (2005). Student and teacher perceptions of learning needs: A cross analysis. Shiken, 9 (2), 8-20. Retrieved on 15 August 2006 from http://www.jalt.org/test/kik_1.htm.

Kusanagi, Y. & Kumazawa, T. (2004). A pilot needs analysis of Rikkyo University freshmen. The Journal of Rikkyo University Language Center, 9, 47-79.

Kuwabara, H., Nakanishi, T., & Komai, K. (2005). Needs analysis of the general English classes. Studies in Communication: Bulletin of the Faculty of Humanities of Ibaraki University, 14, 27-54.

Long, M. H., & Crookes, G. (1992). Three approaches to task-based syllabus design. TESOL Quarterly, 26 (1), 27-56.

Molloy, L. & Newfields, T. (2005). Some preliminary thoughts on statistics ahttp://www.jalt.org/test/mn_3.htmnd background information on SPSS (part 3). Shiken, 9 (2), 2-7. Retrieved on August 13, 2006 from http://www.jalt.org/test/mn_3.htm.

Norris, J. M., Brown, J. D., Hudson, T., & Yoshioka, J. (1998). Designing second language performance assessments. Honolulu, HI: Second Language Teaching and Curriculum Center, University of Hawai'i Press.

Nunan, D. (1988). The learner-centered curriculum. Cambridge: Cambridge University Press.

Richards, J. (2001). Curriculum development in language teaching. Cambridge: Cambridge University Press.

Robson, G. & Midorikawa, H. (2001). How reliable and valid is the Japanese version of the Strategy Inventory for the language learning (SILL)? JALT Journal, 23 (2), 202-226.

Suzuki, S. & Kumazawa, T. (2006). A study on Japanese secondary school students' beliefs about English learning. ASTE Newsletter, 54, 15-27. Retrieved on August 8, 2006 from http://www.bun-eido.co.jp/aste/aste54.html.

Tudor, I. (1996). Learner-centeredness as language education. Cambridge: Cambridge University Press.

West, R. (1994). Needs analysis in language teaching. Language Teaching, 27 (1), 1-19.