Implementation and Evaluation of Evolutionary Connectionist Approaches to Automated Text Summarization

: Problem statement: Text summarization takes care of choosing the most significant portions of text and generates coherent summaries that express the main intent of the given document. This study aims to compare the performances of the three text summarization systems developed by the authors with some of the existing Summarization systems available. These three approaches to text summarization are based on semantic nets, fuzzy logic and evolutionary programming respectively. All the three represent approaches to achieve connectionism. Approach: First approach performs Part of Speech (POS) tagging, semantic and pragmatic analysis and cohesion. The second system under discussion was a new extraction based automated system for text summarization using a decision module that employs fuzzy concepts. Third system under consideration was based on a combination of evolutionary, fuzzy and connectionist techniques. Results: Semantic net approach performs better than the MS Word summarizer as far as the semantics of the original text was concerned. To compare our summaries with those of the well known MS Word, Intellexer and Copernic summarizers, we use DUC’s human generated summaries as the bench-mark. The results were very encouraging. The second approach based on fuzzy logic results in an efficient system since fuzzy logic mimics decision making of humans. Third system showed promising results as far as precision and F-measure are concerned than all the other approaches. Conclusion: Our first approach used WordNet, a lexical database for English. Unlike other dictionaries, WordNet does not include information about etymology, pronunciation and the forms of irregular verbs and contains only limited information about usage. To overcome this limitation, we developed a new text summarizer based on fuzzy logic. As Text summarization application requires learning ability based on activation, we utilize ANN attribute through a connectionist model to achieve the best results


INTRODUCTION
Connectionism is a technical term for a group of related techniques. These techniques include areas such as Artificial Neural Networks, Semantic Networks and a few other similar ideas.
Over the past half a century, the problem of text summarization has been addressed from many different perspective, in various domains and using various paradigms. This study intends to investigate Connectionist architecture for the Text Summarization system, taking into account of existing new developments in adaptive evolving systems. Evolving processes, through both individual development and evolution, inexorably led the human race to our supreme intelligence and our superior position in the animal kingdom.
In this study, we consider the system of an Automatic Text Summarization as Evolving system which learns incrementally through experience in the environment. This study highlights the practical experiences, Connectionist learning environment and new ideas to promote further validations.
Practical experiences: Despite the successfully developed and used methods of Computational Intelligence (CI), such as Artificial Neural Networks (ANN), Fuzzy Systems (FS), evolutionary computation, hybrid systems and other methods and techniques, there are a number of problems while applying these techniques to Text Summarization problem: • Difficulty in preselecting the system's architecture • Catastrophic forgetting • Excessive training time required • Lack of knowledge representation facilities To overcome the above problems, improved and new connectionist and hybrid methods and techniques are required both in terms of learning algorithms and systems learning (Richard et al., 2008;AL-Salami, 2009;Boukerram and Azzou, 2006;Hergli et al., 2005).

Connectionist learning environment: An Evolving
Connectionist System is an adaptive, incremental learning and knowledge representation system that evolves its structure and functionality (Zhijun and Minghong, 2005). Evolving Connectionist System is a CI system based on neural networks, but using other techniques of CI that operate continuously in time and adapt their structure and functionality through a continuous interaction with the environment. Figure 1 explains a typical connectionist learning environment.
This study describes three approaches to Automated Text Summarization using connectionist statures based on: • word net, an online dictionary based on Semantic Nets (SN) • Fuzzy logic • Evolutionary connectionist and fuzzy techniques This study is organized as follows: Introduction to the domain immediately follows details of implementation details of all the three summarizers specified above. This is followed by Results, Discussion and Acknowledgement. Authors have surveyed the current text summarization approaches (Afantenos et al., 2005;Ledeneva et al., 2008) their advantages and disadvantages and, with the goal of identifying summarization techniques most suitable to generic text summarization. Precision/recall schemes, as well as summary accuracy measures which incorporate weightings based on multiple human decisions, are suggested as particularly suitable in evaluating generic summaries.

MATERIALS AND METHODS
Text summarization based on word net, an online Dictionary based on Semantic Nets (SN): Methods for text classification and information retrieval have been recently presented making use of the word net ontology. Generally, this methodology requires statistical induction of synset clusters and entails costly training of specific key domains. The present study word net is rich enough to obtain useful results in text categorization and summarization without training the tagged corpora.

Part of Speech (POS) tag and dependency tree generation:
We use the Stanford POS tagger to identify nouns and adjectives in the sentences as shown in Fig. 2. The Stanford POS tagger tags Nouns and Noun Phrases as NN, NNP, NNS and adjectives as JJ. Furthermore, a sentence could contain more than one Noun or Noun Phrases (features) and Adjectives (opinions). Thus, we need to determine the Noun that a particular Adjective modifies i.e. the feature about which a certain opinion has been expressed. For this purpose we used the Stanford Parser to generate the parse tree of a sentence and extract typed dependencies among the words of a sentence. The typed dependencies provide a description of grammatical relationships between the words of sentence.

Summarization algorithm module:
After the semantic grading of the nouns and verbs, also called nuclei, has been done, keywords among the nuclei are identified these keywords are nuclei having a semantic grading or polysemy count of <=5. Also, modifiers i.e., adjectives and adverbs, having a semantic grading of <=5 are picked, given that they relate to keywords. A separate algorithm is developed to determine which modifiers apply to a nucleus. After all the keywords have been determined, keyword counts of each and every sentence, the semantic unit of summary, are determined. Then, the semantic appreciation of each sentence i.e. modifier effect on nuclei is determined. Using these two criteria and considering proper nouns, sentences for the summary is picked.
Text summarization based on fuzzy logic: We next focus on the second system that is an automated text summarization system based on statistical approach using fuzzy logic over some significant text features. Significant text features considered in the design of the proposed system are word similarity among sentences, word similarity among paragraphs, iterative query score, format based score, numerical data, cue-phrases, term weight, thematic features and title features. The extracted text features are then mapped into the fuzzy logic to score each sentence. The summary is extracted from the document based upon the score of each sentence. The proposed automated text summarization system consists of five components: Preprocessing, feature extraction, fuzzy logic scoring, sentence selection and assembly. The system architecture is illustrated in Fig. 3.

Preprocessing:
Preprocessing is the first component of the system with three different phases: sentence segmentation, removing stop words and, stemming. After applying preprocessing techniques, individual sentences and their unique ID are obtained from the text document: • Segmentation process is achieved by finding out the delimiter ("." full stop) so that, the sentences in the document are separated • Stop words (Pant et al., 2004) are detached from the document during the feature extraction step since they are considered as unimportant and contain noise. Stop words are predefined and are stored in an array and the array is utilized for comparison with the words in the provided document • Word stemming (Lovins, 1968) converts every word into its root form. Word stemming is practically removing the prefix and suffix of the specified word which in turn becomes applicable for comparison with other words Feature extraction: The document after preprocessing is subjected to feature extraction by which the properties of the sentences are extracted to score the sentence. The significant text features considered in the proposed system are: Word similarity among sentences: A sentence is given a high score when the terms or the words in it occur in more number of other sentences in the document. Each sentence is segmented into individual words; the segmented words are searched in the other sentences of the given document. The number of other sentences in which a given word has occurred is termed as the occurrence count of the word. The occurrence count of all the individual words in the sentence is summed up to get the Sentence Occurrence Count (SOC). The score for the feature, word similarity among sentences is calculated as the ratio of the sentence occurrence count of the given sentences to the maximum sentence occurrence count in the document: Word similarity among paragraphs: The feature is extracted from the whole paragraph rather than from individual sentences. Thus all the sentences under a single paragraph will get the same score. This feature is analogous to the word similarity among sentences and the Paragraph Occurrence Count (POC) of a given paragraph is defined as the number of paragraphs in the document that contains the same terms or words as the given paragraph: Iterative query score: The score corresponding to this feature is accomplished by three phases: • Initial keyword identification: The top 'n' frequent words are selected as initial keyword set • Scoring sentences based on iterative query Query is nothing but searching for a keyword in the given document and retrieving those sentences that contains the keyword. A tag named count will be added to all the sentences in the document which keeps track of the number of appearance of the sentences in the query result of all iteration. In every iteration, the tag count of each sentence will be updated. The iteration stops when predefined number of loops is executed or if there is no change in the keyword list. The sentence score for this feature is the ratio of the count to the total number of iterations: where count is the number of iterations in which the sentence has occurred.
Format based score: In many of the documents the importance of the sentences or headings is indicated by expressing the text in different text format e.g., Italics, Bold, underlined, big font size and more. This feature is some what specific to a single sentence and do not depend upon the whole document. By considering the format of the words in the text we can assign a score to the sentence: Number of words in the sentence with special format FB _ f (s) Total number of wordsin the sentence = (4) Numerical data: The numerical data in the document generally brings about some important stats of the core idea of the document and thus the sentence with numerical data can reflect the intention of the document and may be selected for the summary. The score for this feature is calculated as the ratio of the number of numerical data that occur in sentence over the sentence length: Length of numerical data in the sentence NU _ f (s) Sentence length = (5) Cue-phrases: Generally phrases such as "in summary", "in conclusion" and superlatives such as "the best", "the most important", "according to the study", "hardly" can be good indicators of important content of a document (Zadeh, 1965). The sentences that contain cue words/phrases are given a higher score than those not containing them. If the sentence contains the cue phrases the sentence gains a score calculated by: No. of cue words in the sentence CP _ f (s) Total no. of cue phrases in the documents = (6) Term weight: The term weight of all the terms or words in the given document is calculated and stored for all the words. The term weight for each word is given by the following formula: where is the frequency of a particular term that appears in the document and is given by: where, t is a term in the sample document is the total number of sentences in the document and is the number of sentences in which t occurred. The summation of the calculated term weight of all the terms in a sentence gives the sentence weight in the document: Where: W_S = Sum of term weights in a sentence n = Number of words in the sentences W i = Weight of the ith word in the sentence S The score of a sentence is calculated as the ratio of the sentence weight to the maximum sentence weight in the given document: Title features: A sentence is given a higher score if it contains the words that occur in the title. The sentence which contains the words that occur in title may give what the document is intended to express. The score of a sentence for this feature can be calculated as the ratio of the number of words in the sentence that occur in title to the total number of words in the title.
Every sentence in the document along with its ID has a feature vector with nine fields for the aforesaid nine features. All the features will have the value range between 0 and 1.

Fuzzy logic scoring: Fuzzy logic was introduced by
Zadeh in the late 1960s (Zadeh, 1965) and is considered as the rediscovery of multi-valued logic. In fuzzy logic, the truth values of the variables can take any value in the range 0-1 (e.g., 0.23), in contrast to Boolean logic, in which variables can be either 1 or 0. Triangular membership function and fuzzy logic are used to score a sentence based on the above extracted text features. The Fuzzy logic system consists of four parts: • Fuzzifier • Rule base • Inference engine • Defuzzifier Fuzzifier: A fuzzifier converts the input feature values into linguistic values (Very Low, Low, Medium, High and Very High) using the membership function. The linguistic value denotes a fuzzy set (e.g., Low) to which a given sentence feature belongs. Fuzzy set FS can be defined as set of ordered pair: Where FS-> = {Very Low, Low, Medium, High, Very High} x->[0,1] x = Denotes a text feature of the given sentence f(x) = The Triangular membership function of a fuzzy set given by: where a, b, c are characteristic parameters of a fuzzy set S.
The linguistic value of a given sentence can be determined using the support of each fuzzy set. The support (supp) of a fuzzy set FS is nothing but the list of sentences which give non-zero values for the membership function of FS: where is the set of sentences in the document? It is enough to check all the fuzzy set for the given sentence to determine the linguistic value of a sentence. The linguistic value is the name of the fuzzy set in whose support list the given sentence occurs. There is a possibility that a sentence may belong to more than one fuzzy set, in this case the sentence is considered to belong to the fuzzy set whose membership function gives minimum value for the given sentence.

Rule base:
The most important procedure in any fuzzy system is defining the fuzzy IF-THEN rules. A rule consists of two parts antecedent and consequent. Antecedent is the possible input feature values and consequent is the inference of the rule which determines whether the sentence is important, average or unimportant based on the input. Sample of fuzzy rule is given below: IF (Word co-occurrence among sentence is H) and (Word co-occurrence among paragraph is VH) and

(Iterative query score is H) and (Format based score is M) and (No. cue-phrases is H) and (Term weight is VH) and (Thematic feature is VH) THEN (Sentence is important)
Inference engine: The Inference engine compares the fuzzy input obtained from the fuzzifier with the Knowledge base and decides the importance of a sentence. The output of inference engine is one of the linguistic values from the set {Unimportant, Average and Important}.

Defuzzifier:
The linguistic values obtained from the inference engine are converted into crisp values by the defuzzifier. The crisp value denotes how close the sentence is to the given linguistic value.

Sentence selection and assembly:
The selection of a sentence consists of two steps: (1) determining the number of sentence to be in the summary based on compression rate and (2) extracting the appropriate sentences for the summary. The number of sentences N to be placed in the summary is calculated as:

Compression rate
No. of sentences the summary (N) 100 Total no. of sentences in the document = × (16) Sentence extraction is accomplished by first arranging the sentences in descending order based on the crisp output value from defuzzifier and the top N sentences are selected for the summary. To obtain clear and logical summary, sentences that are selected to be included in the summary are sequentially ordered based on the order of the reference number or unique ID of the sentence.
Text summarization based on evolutionary connectionist and fuzzy techniques: In our prior work , 2009c, we described an automatic text summarization system using fuzzy logic. Authors aim to introduce an efficient and effective system for automated text summarization that combines evolutionary, connectionist and fuzzy techniques. Figure 4 depicts the proposed system architecture for text summarization based on evolutionary, connectionist and fuzzy techniques.
The proposed automatic text summarization system consists of the following components: • Preprocessing • Feature extraction

Evolutionary Programming (EP) model:
The preprocessed sentences is subjected to feature extraction process so that, the feature vector is computed for each sentence. Evolutionary Programming (EP) module generates large number of feature vectors (chromosomes) iteratively utilizing cross over and mutation operators subsequently, fuzzy logic is employed on the chromosomes and it returns the fuzzy score for each chromosome. Then, the chromosomes with their fuzzy score are fed to the neural network for training.
Connectionist model: Authors describe here, the connectionist model used in the proposed approach for automatic text summarization. Normally, neural networks are a great deal the most frequently used connectionist model at present. A lot of research using neural networks is made under the more common name "connectionist". Here, we have used the Multi-layer Perceptron Neural Network (MLPNN) A multilayer perceptron is a feed forward artificial neural network model that has at least one layer in-between the input and the output layer. A neural network MLP couples, through functions and weights, certain variables (called inputs) with certain other variables (called outputs) (Lahoz and Miguel, 2006). The neural network used in the proposed system is configured with a nine input, hidden and one output layer. The configurations of the network used for our approach is shown in the Fig. 5.

Structure of multi-layer perceptron neural network:
Training phase-The back-propagation algorithm can be utilized successfully to train neural networks; it is extensively accepted for applications to layered feedforward networks, or multi-layer perceptrons (Aliruliyev, 2009). In order to train the neural network optimally, the input layer is an individual (feature vector) obtained from the EP and the target output is the fuzzy score of the relevant individuals. So, for training the neural network, we make use of evolutionary and artificial intelligence techniques (Kursk et al., 2006).
Testing phase-In testing phase, feature score of every sentence in the document is computed. The computed feature score is applied to the trained network that returns the final score of every sentence presented in the input text document. Based on the computed score value, the coherent and correctly-developed summary is generated for the given input text document input, we make use of EP model, which is based on the genetic operators such as, cross over and mutation.
The first step is the generation of an initial population for evolutionary process. The feature vector of a sentence is known as chromosome (candidate). The set of chromosomes are obtained for every sentence in the text document. Then, by making use of evolutionary concept, more candidates are generated from an initial population. In order to generate large number of candidate sets, we have used the genetic operators such as cross over and mutation (Haupt and Haupt, 1997).
Selection: Two random integers are generated within the size of the population. Then, two chromosomes corresponding to the generated number are selected from the initial population.

Crossover:
The crossover operator is applied on the selected two candidates and this produces two individuals newly. Here, we have used the single point cross over.

Mutation:
The obtained new set of individuals is then fed to the mutation operator. To have a better exploration of the search space, mutation operator is carried out. Again, we obtain two individuals newly from the single point mutation operator.

Termination:
The population is updated with four new set of individuals. Again, the selection, crossover and mutation operators are performed iteratively.
Sentence selection and assembly: Two important steps are involved in the selection process of a sentence (1) determining the number of sentence that must be present in the summary based on compression rate and (2) appropriate sentence extraction for the summary. The number of sentences to be placed in the summary is calculated as: Where: N s = Total number of sentences in the document C = Compression rate Based on the crisp output value from defuzzifier, sentence extraction is attained by arranging the sentence at first in the descending order and thereby the top sentences are chosen for the summary. A summary has to possess a comprehensible structure and should be presented in a logical manner. On the basis of the order of the reference number or unique ID, the sentences are sequentially ordered to get the final summary.

RESULTS
The experimental results and analysis of the proposed automatic text summarization system is presented here. The proposed system is implemented in MATLAB (MATLAB 7.8). We have used DUC 2002 dataset in the proposed system for generating the single document summary. DUC 2002 dataset contains documents on different categories and extractive summary per document.  ID  F1  F2  F3  F4  F5  F6  F7  F8  F9  S1 1  The experimentation is performed in two different phases namely, training phase and testing phase. Training phase: In the proposed system, as a training data, we have taken 100 sentences from the DUC 2002 dataset (Document No: AP880916-0060, AP900322-0112, AP890607-0067 and LA122190-0149). And then, we apply the preprocessing and feature extraction techniques on the training data so that, we obtain the 100 feature vectors. The sample feature score of the text document (Document No. AP880314-0110) is shown in Table 1.
The feature vectors chosen from the EP model and their corresponding fuzzy score are used for better training of the neural network. We have used the Multi Layer Perceptron Neural Network which contains nine input layer and one output layer. Testing phase: The input document is taken from the dataset and the preprocessing and feature extraction techniques are applied on the input document. The feature score obtained for the input document (Document No. LA080890-0078) is given in Table 3.
The feature score is then directly applied to the trained neural network which returns the sentence score for every sentence in the document. The sentence score obtained from the neural network for the input document is given in Table 4. Finally, the salient sentences are extracted by inputting the compression rate.
Evaluation measure: The performance of the proposed approach is evaluated using precision, recall and Fmeasure. Precision evaluates the proportion of correctness for the sentences in the summary whereas recall is utilized to evaluate the proportion of relevant sentences included in summary. For precision, the higher the values, the better the system is in omitting irrelevant sentences. Conversely, the higher the recall values the more successful the system would be in fetching the relevant sentences. The weighted harmonic mean of precision and recall is called as F-measure: Performance evaluation: The performance of the proposed system is evaluated on the summary available in the DUC 2002 dataset using the evaluation measures described above. We have taken four documents from the dataset, D 1 (AP880310-0062), D 2 (AP880622-0184), D 3 (AP880816-0135) and D 4 (FT923-5835). Then, we generate the single document summary for these four documents using the proposed system. For experimentation, the summary is generated for different compression rate and the generated summary is evaluated on the extractive summary provided in the dataset using the evaluation measures such as, precision, recall and F-measure.   Sentence ID  F1  F2  F3  F4  F5  F6  F7  F8  F9  S1 0  The computed evaluation measures for compression rate C = 40 is given in Table 5 and their corresponding graph is shown in Fig. 6. Similarly, the performance graph is plotted for compression rate C = 50 and C = 60, which is shown in Table 6 and 7 and Fig. 7   The authors compare the average precision, recall and F-measure score between Copernic, Intellexer, General Statistic Method (GSM), Microsoft Word 2007 summarizer systems and the three summarizers developed by the authors 2009c, 2009c. The results shown in Table 8 show Approach one reaches the average precision of 0.70000, recall of 0.76666 and F-measure of 0.65555, the second summarizer achieves the average precision of 0.83051, recall of 0.79000 and F-measure of 0.83 and connectionist summarizer achieves precision of 1, recall 0.77 and F-measure of 0.87.

DISSCUSSION
We have developed automatic text summarization system with three different approaches. The purpose was to implement and evaluate existing connectionist methods and adopt the best suited for the domain of text summarization process. The experimental results show that the third approach, which combines EP model, Artificial Neural Network (ANN) and fuzzy logic suits the said domain appropriately. Here, we have used nine different features for feature extraction phase. Then, the feature vectors are iteratively generated by making use of EP model. Subsequently, the feature vectors are given to the fuzzy logic system so that, the fuzzy score is calculated. The feature vector and their relevant fuzzy score are utilized as a training parameter for training the neural network. In the testing phase, the features extracted from the input text document are given to the trained network that provides score for every sentence in the input document. Finally, we extract the relevant sentences from the input text document in accordance with their sentence score. We have used DUC 2002 dataset to evaluate the summarized results based on the measures such as Precision, recall and F-measure. The experimental results showed that the proposed summarization system effectively summarizes the text documents.

CONCLUSION
Since a lot of interesting work is being done far from the mainstream research in this field, we have chosen to develop approaches to Text Summarization that we found relevant to future research, even if they focus only on small details related to a general summarization process and not on building an entire summarization system. The results obtained, suggest that the future of this research area heavily depends on the ability to find efficient ways of automatically evaluating these systems and on the development of measures that are objective enough to be commonly accepted by the research community.