Pain assessment in surgical patients with impaired cognition
Abstract
Pain is considered the 5th vital sign and its measurement/assessment and records are required and must be systematic. Ineffective pain management involves complications in clinical status of patients, longer hospitalization times and higher costs with health. In the surgical patient with impaired cognition, hetero measurements should be made, based on behavioural and physiological indicators. To determine the efficacy and efficiency of the Observer Scale, the Abbey Pain Scale and Pain Assessment in Advanced Dementia (PAINAD). This is an applied, non-experimental, quantitative, descriptive and analytical research. The data collection instrument consisted of patients’ sociodemographic and clinical data, the Observer Scale, the Abbey Pain Scale (Rodrigues, 2013) and PAINAD (Batalha et al., 2012). We assessed pain at an early phase and 45 minutes after an intervention for its relief. The sample is non-probabilistic for convenience, consisting of 76 surgical patients with impaired cognition, admitted to the surgery services of a central hospital, aged between 38 and 96 years. There was a positive correlation between the results of the three scales, most evident in the initial evaluation. Pain intensity in the same patient is higher when assessed with PAINAD (OM = 2.16) and lower when assessed with the Observer Scale (OM = 1.78). The most effective and efficient scale is PAINAD. Due to the small sample size, we suggest confirmatory studies so that the results can be generalized.
Keywords: Keywords Acute pain, cognition, general surgery, pain postoperative, pain measurement
Introduction
This empirical study issue we identified relates to the fact that there has not been a correct assessment, management and registration of pain of surgical patients with impaired cognition. There are properly validated scales for the Portuguese population and their use is mandatory. This is the Numerical Scale, recommended by the Order of Nurses and the General Health Directorate. In addition, it is known to all nurses that pain is considered the 5th vital sign and as such should be assessed often and systematically and the results should be registered in the same way as other vital signs.
In our working context, pain assessment in the specific group of surgical patients with impaired cognition is performed only by means of the Observer Scale, which is a hetero-assessment, one-dimensional scale, based solely on the evaluator’s opinion. It seems insufficient and too reductive.
The state of the art on this issue was studied. Some authors have published papers on this topic and a number of pain assessment scales have been suggested. However, a question arises: of the existing pain assessment scales, which is the most appropriate for use in patients with impaired cognition?
Therefore, looking at the selected measuring instruments, validated for the Portuguese population, for this study, we pondered: what is the effectiveness of the pain assessment scales (the Observer Scale, the Abbey Pain Scale and PAINAD) used in patients with impaired cognition, admitted to general care and intermediate care surgery units? Does pain intensity, subjectively assessed in surgical patients with impaired cognition, differ depending on the scale used? Which scale better evaluates the prevalence of pain in surgical patients with impaired cognition?
From this perspective and in order to attempt to answer these questions, the following objectives are proposed: to determine the effectiveness of pain assessment scales (the Observer Scale, the Abbey Pain Scale and PAINAD) used in patients with impaired cognition admitted to general care and intermediate care surgery units; to compare the intensity of subjectively assessed pain in surgical patients with impaired cognition obtained by different scales; to compare the prevalence of pain in surgical patients with impaired cognition assessed by the three scales.
The data collection instrument was then built, incorporating three of these scales: the Observer Scale (already available in the institution’s computer programme), the Abbey Pain Scale (Rodrigues, 2013) and the Pain Assessment in Advanced Dementia (PAINAD) (Batalha et al., 2012). Having complied with the ethical requirements the circumstances demand, the data were collected, processed and analysed.
Theoretical Framework
Pain – the 5th vital sign
Pain is a subjective perception of a universally known phenomenon (Kazanowski & Laccetti, 2005, p. 3). It is defined by the International Association for the Study of Pain (2002, as cited in Metzger, Muller, Schwetta & Walter, 2002, p. 5) as
According to Kazanowski and Laccetti (2005, p. 3), pain is characterized by various objective signs and symptoms. However, these vary from person to person. It should, therefore, be carefully evaluated for proper planning of interventions.
Fillingim (2000, as cited in Cardoso, 2013, p. 8) draws attention to neurophysiological, clinical and psychosocial differences in the perception of pain stimuli in terms of gender. The neurophysiological differences mentioned are lower pain threshold, tolerance and sensitivity in women; higher sensitivity to opioids and adjuvant medications in women. The clinical differences relate to the fact that women describe pain as more intense, more frequent and more prolonged.
The psychosocial differences mentioned are differences in expression of pain; differences in dealing with pain; differences due to the interaction of the male/female patient with health professionals; submedication of pain, which is more common in women than in men.
In 1993, Bendelow published a study which concluded that men take longer to admit that they feel pain and, when they do, they receive more attention and care by health professionals. Also, according to the same study, women tend to have more psychogenic pain. (Cardoso, 2013, p. 9)
Pain assessment
Cardoso (2013, p. 26) considers pain difficult to quantify, given its multidimensional nature, i.e. the patient’s physical pain is affected by various factors, such as psychological, social and spiritual aspects.
However, the importance of assessing and managing pain is recognized by governments, so that in 1999 the General Health Directorate established the Working Group for Pain. Working in collaboration with the Portuguese Association for the Study of Pain, they created the National Plan to Fight Against Pain, which was approved in March 2001. It was a pioneering work, since when it was created, only France had a similar plan among European countries. Here a number of guidelines for pain control were drafted as well as organizational models for pain management in hospitals.
As it is a subjective vital sign, pain assessment should be performed by individuals themselves; that is, it should be a self-assessment whenever possible.
The General Health Directorate issued a Regulatory Circular (Normative Circular No. 9/DGCG of 14/06/2003) with precise indications concerning the systematic recording of pain intensity in all patients in all health care services at the same place where other vital signs are registered, suggesting for this purpose some scales for use in conscious and cooperating patients over the age of three years.
Also in 2008, the Order of Nurses made pronouncements on the subject of pain, issuing a Good Practices Orientation Guide on 14 June, which was precisely the National Day Against Pain. In this guide, there are pain assessment and management principles as well as recommendations for professional practice based on available scientific evidence and expert opinion. Various scales for certain specific groups of patients are also suggested.
The surgical patient
Kazanowski & Laccetti (2005, p. 85) are of the opinion that, despite fears related to the risks of surgery itself, one of a person’s main fears when subjected to surgery is the fear of pain.
In emerging surgery, treatment of pain is essentially postoperative. In elective surgery, there individuals must be prepared in advance, in the preoperative period, where they are educated on a variety of pain control techniques.
“The objectives related to surgical pain involve treatment of pain preoperatively, preventing sensitivity or trans-operative pain and the prevention or relief of postoperative pain during recovery and rehabilitation.” (Kazanowski & Laccetti, 2005, p. 87)
Pain assessment in the surgical patient
Pain assessment and management are of the utmost importance in the surgical patient, since it is a key symptom of a clinical condition. As stated by Pimenta, Santos, Chaves, Martins and Gutierrez (2001, pp. 181-182), the impact of ineffective pain control postoperatively is immense and harmful to patienta and the evolution of their clinical condition. This may include difficulty in deep ventilation and consequent reduction in elimination of respiratory tract secretions, which can lead to atelectasis and respiratory infections; respiratory, hemodynamic and metabolic changes that predispose the patient to cardiovascular instability, increased consumption of energy and protein and reduced ventilatory volume. It may also be associated with cardiac arrhythmias, atelectasis, pneumonia and protein-caloric depletion, among others; reduce movements and early ambulation, favouring the onset of deep vein thrombosis, especially in elderly patients and those undergoing major surgery. It can interrupt sleep, which may result in greater physical stress, fatigue and less motivation to cooperate with the treatment. It also contributes to higher hospitalization time and consequently higher health costs.
However, pain assessment continued to cause great difficulties. Batalha (2005, p.166) states that “pain assessment is an ignored activity, forgotten or performed unreliably (imprecise),” which “seriously compromises quality of care.”
Patients with impaired cognition
Kazanowski and Laccetti (2005, p. 185) remark on a number of changes in the thinking process, one of which is the confusion of the elderly. There can be several causes, such as organic causes, Alzheimer’s disease, organic brain syndrome, chronic or acute episodes of hypoxia or sleep deprivation, the side effects of medication, being away from their familiar surroundings or routine.
According to the International Classification for Nursing Practice (ICNP) version β2, nursing diagnoses used to characterize these patients are: altered consciousness, compromised cognition, confusion and agitation.
Pain assessment in the surgical patient with impaired cognition
If assessment is difficult in most patients, in cognitively impaired patients, it becomes even more difficult and even impossible at times, consequently hindering its diagnosis and control. Rothrock (2008, p. 260) suggests “special attention” for “patients who find it difficult to report their pain” (those
with impaired cognition, psychosis, severe emotional imbalance, children, the elderly, those who do not speak the same language, those whose educational level and culture differ from the health team). Therefore, “appropriate scales for the physical, emotional, and cognitive development of the patient” should be used.
Kazanowski and Laccetti (2005, p. 186) say that “there may be no evidence of verbal expression of pain or requests for relief.” We must therefore be alert to physical indicators of increased pain, such as tachycardia, tachypnea, change in blood pressure levels, frequent position changes, behaviours of protection, rigidity in the painful area, pallor and diaphoresis and behavioural changes.
In these cases, the scales to be used may not be the same since verbal communication in these patients is compromised. Hetero-assessment scales which including behavioural and/or physiological aspects should be used.
In order to compare the scales suggested for assessment of acute pain (referring to surgical patients) in patients with impaired cognition a brief summary about the Observer Scale, the Abbey Pain Scale and PAINAD, validated for the Portuguese population (Order of Nurses, 2008, pp. 37-39 and Rodrigues, 2013, p. 106) follows as these were used in this study.
The Observer Scale is also a qualitative, one-dimensional hetero-assessment scale, which can be used in patients with an altered state of consciousness or impaired cognition. It is adapted from the Hitchcock Pain Scale. It is a graded scale, which describes patients’ pain by category: no pain, mild pain, moderate pain, severe pain, and maximum pain.
Rodrigues (2013, p. 81) developed a study on “Pain Assessment in Oncological Patients in Palliative Care Who Are Unable to Communicate,” validating the Abbey Pain Scale (to assess pain in people who are unable to communicate), creating the Portuguese version. This scale consists of six assessment items which correspond to non-verbal pain indicators: vocalisation, facial expression, change in body language, behavioural changes, physiological changes and physical changes. Each item is given a score according to the following: “absent=0”, “mild=1,” “moderate=2” and “severe=3.” The sum of the six items, “total pain,” will give a final score between 0 and 18, such that it corresponds to levels of pain as follows: from 0 to 2 points – no pain; from 3 to 7 points – mild pain; 8 to 13 points – moderate pain; 14 to 18 points – severe pain.
Pain Assessment in Advanced Dementia (PAINAD) is indicated for the elderly with dementia. The Portuguese version includes 5 categories or indicators to be assessed, which are as follows: breathing regardless of vocalisation, negative vocalisation, facial expression, body language and consolability. The final score ranges from 0 to 10, with higher values indicating greater intensity of pain. (Batalha et al., 2012, p. 10)
Methodology
The choice of this topic emerged precisely from the need to properly assess pain in surgical patients with impaired cognition. This is often found to be underestimated and undervalued, which translates into more confused and agitated patients with more complications. In our work context, we think it is very important to use a scale to assess these patients’ pain, so that we can tailor their treatment and consequently facilitate their recovery.
A qualitative scale, the Observer Scale, is currently available in the computer programme in the institution, but it is not used regularly and systematically. Moreover, this programme only reflects the opinion of the observer in relation to the intensity of pain experienced by the patient, neglecting other aspects often seen as changes manifested in other vital signs (blood pressure, heart rate, respiratory rate, oxygen saturation) as well as behavioural changes (patients who react to the sensation of pain, appearing confused and/or agitated).
Therefore, the need to find out if there are other scales that are appropriate for this population and which involve other indicators appeared and if so, to try them.
After researching the literature, two other scales were found. They are based on physiological and behavioural indicators: the Abbey Pain Scale (Portuguese version) and Pain Assessment in Advanced Dementia (PAINAD).
In view of these new instruments, we pondered: what is the effectiveness of the pain assessment scales (the Observer Scale, the Abbey Pain Scale and PAINAD) used in patients with impaired cognition, admitted to general care and intermediate care surgery units? Does pain intensity, subjectively assessed in surgical patients with impaired cognition, differ depending on the scale used? Which scale better evaluates the prevalence of pain in surgical patients with impaired cognition?
From this perspective and in order to attempt to answer these questions, the following objectives are proposed: to determine the effectiveness of pain assessment scales (the Observer Scale, the Abbey Pain Scale and PAINAD) used in patients with impaired cognition admitted to general care and intermediate care surgery units; to compare the intensity of subjectively assessed pain in surgical patients with impaired cognition obtained by different scales; to compare the prevalence of pain in surgical patients with impaired cognition assessed by the three scales.
The aim of this study is to identify the most appropriate scale to apply to surgical patients with impaired cognition.
In order to answer the questions and aims outlined, we develop an applied, non-experimental, quantitative and descriptive-analytic study.
The sample is non-probabilistic for convenience, consisting of surgical patients with impaired cognition hospitalised in the surgery service in a central hospital in the central region of the country.
In selecting the sample, the following was used as inclusion criteria: they must be a surgical patient, they must have been hospitalised in the following units: Surgery 1A, Surgery 1B, the Surgical Patient Monitoring Unit, Surgery 2A, Surgery 2B and the Surgical Monitoring Intermediate Care Unit of and include at least one of the following diagnoses in their (computerized) care plan: altered consciousness, impaired cognition, confusion and/or agitation.
The data collection instrument is a questionnaire that incorporates sociodemographic variables and the following pain scales: the Observer Scale, the Abbey Pain Scale – Portuguese version and Pain Assessment in Advanced Dementia (PAINAD) – Portuguese version.
In the first part of the instrument, the patient’s data are identified, including age, gender, hospitalisation services, medical diagnosis, surgery, personal history and nursing diagnoses which justify cognitive impairment.
The scales mentioned above can be found in the second part, i.e. those which are recommended for people who do not know/cannot effectively communicate their pain (acute pain).
the questionnaire ends with issues related to other pain characteristics, in particular, its anatomical location, duration, classification and type.
With regard to the psychometric study, the Abbey Pain Scale obtained a reasonable global alpha value in the first assessment (α = 0.755). However, in the second assessment it tends to be weak (α = 0.638). As for PAINAD, its internal consistency is good in the initial assessment (α= 0.836) but weak in the following assessment (α = 0.679).
The study took place in the surgery services and respective intermediate care unit of a central hospital in the centre region of Portugal from January to April 2015.
The nursing team of the surgery services proceeded to complete the questionnaire whenever one of the nursing diagnoses possessed the criteria required for inclusion and 45 minutes after the implementation of a pain control intervention.
Results
Table 1 shows the Pearson correlation matrix with the scores of the three scales for both assessments. As noted, the correlation coefficients for both assessments are positive and significant, which allows us to say that the best scores obtained for a scale correspond to higher scores on the scale to which it relates.
Analysing the results of the initial assessment, we note that the lowest correlation is obtained in association (r = 0678) between the Abbey Pain Scale and the Observer Scale and the highest (r=0.842) between the Abbey Pain Scale and PAINAD with 70.89% explained variance.
In the subsequent assessment, we noted the same trend, albeit with a slightly lower correlation coefficients and variability for both scales of 46.78%.
These results show to a certain extent that the Observer Scale is the least effective in pain assessment.
We compared the pain intensity ratings of the three instruments applied to the same subject in order to verify their effectiveness. The Friedman test showed that pain intensity in the same subject is higher when assessed with PAINAD (OM=2.16) and lower when assessed with the Observer Scale (OM=1.78) with statistical differences observed between the initial assessments of the 3 scales (x2=6.821, p=0.033). (see Table 2)
In order to locate the differences, the Wilcoxon test was performed, and its results are shown in table 3.
From the first test, comparing the scores from the Abbey Pain Scale with the Observer Scale, we found that the pain intensity assessed by the Abbey Pain Scale was higher than by the Observer Scale but without statistical differences. Between the scores of the Observer Scale and the PAINAD scale, we found better efficacy in the latter with statistical significance. Finally, between PAINAD and the Abbey Pain Scale, it is worth emphasising once again the effectiveness of PAINAD but without statistical significance. We thus find that these two scales measure the same constructs, albeit there is a tendency for PAINAD to obtain higher scores.
Similarly, we wanted to verify the effectiveness of the scales in the subsequent pain intensity assessment. Table 4 reports the results of the Friedman test, verifying that the pain intensity assessed by the Abbey Pain Scale is higher than the others and the lowest was the one assessed by PAINAD with statistical significances between them (x2=58.988; p=0.000)
Having compared the scales two at a time, we found that, in general, pain intensity is higher when assessed by the Abbey Pain Scale relative to the other scales with statistical differences, with the exception of pain intensity assessed by PAINAD vs the Observer Scale.
In short, the Abbey Pain Scale was shown to be more effective in the subsequent assessment countering what had occurred in the initial assessment. (see Table 5)
Discussion
The sample in our study consists of 76 subjects, aged between 38 and 96 years, with an average age of 74.06 years, of which 29 are female and 47, male. Of the respondents, 65.5% of the assessments were performed on surgical patients admitted to Surgery 1 (Surgery 1A, Surgery 1B and Surgical Patient Monitoring Unit) and 34.5% were admitted t0 Surgery 2 (Surgery 2A, Surgery 2B and the Surgical Monitoring Intermediate Care Unit).
As for applying the scales, in the first assessment Rodrigues (2013, p.75), in his study on applying the Abbey Pain Scale to cancer patients in palliative care units, obtained an average pain score ranging between 3.54 (±3.258 SD) and 3.68 (±3.344 SD), corresponding to slight pain. In our study, this value was 6.74 (±4.22 SD), also corresponding to slight pain. in the second assessment, the average pain score is lower in our study (M=2.63 ±2.70 SD), when compared to Rodrigues (2013, p. 76), whose mean values ranged between 3.59 (±2.721 SD) and 3.87 (± 2.607 SD), all of which represent mild pain.
With regard to pain intensity assessed by the Observer Scale, the highest percentages of people with intense pain (12.8%) and maximum scores (4.3%) were found with males, contrary to what Filingim (2000, as cited in Cardoso, 2013, p. 8) referred saying there is a lower threshold, tolerance and sensitivity of pain in women. The highest percentage of women (44.8%) was found to have moderate pain. This author also states there is higher sensitivity to opioids and adjuvant medications in women. However, in our sample the reassessment after the pain relief intervention, showed very similar percentages of men and women.
The assessment results with the Abbey Pain Scale by gender are in line with those obtained in the previous scale as well, and also contrary to the above author. Initially, 12.8% of men were found to have severe pain, 31.9% moderate pain and 44.7% mild pain. In women, 27.6% had moderate pain and 17.2% mild pain. Regarding response to pain relief interventions, the percentages are also similar for both genders.
As for pain intensity assessed by PAINAD and as it is a qualitative classification, therefore subjective in nature, both in the initial assessment and the subsequent assessment after intervention, the percentages for both genders do not differ much from each other, which does not corroborate with the above author either.
When comparing the averages of the results of the initial assessments of the 3 scales, converted to percentages, we see that they are identical (32.57% with the Observer Scale, 37.43% with the Abbey Pain Scale and 39.74% with PAINAD). However, the Friedman test shows statistically significant differences (x2=6.821, p=0.033) between PAINAD and the Observer Scale by the Wilcoxon tests that followed.
In the subsequent assessments, however, the results differ because the average with the Observer Scale is 4.93%, with the Abbey Pain Scale, 14.62% and with PAINAD, 7.10%. In this case, the Friedman test showed highly significant differences (x2=58.988; p=0.000), between the Abbey Pain Scale and the Observer Scale by the Wilcoxon tests (Z=-5.498; p=0.000) and between PAINAD and the Abbey Pain Scale (Z=-5.074, p=0.000).
Moreover, this information is in line with the results of the correlations between the three scales, whose strength is greater in the initial assessment with correlational values ranging between (r=0.678 and r=0.842) than in the subsequent assessment (r=0.569 and r=0.684).
Conclusions
Pain is a subjective symptom which requires specific care in its assessment. In surgical patients, this assessment is important, since, as mentioned in the theoretical framework, poor management of perioperative pain causes many complications in patients themselves, as well as greatly increasing health care costs.
The issue dealt with in this work emerged from the need felt in clinical practice. Its main aim was to identify the most effective scale to be used to assess pain in surgical patients with impaired cognition.
Through the data collected in the surgery services in a hospital in the central region of the country (general care wards and intermediate care units), we were able to determine that the pain in surgical patients with impaired cognition is real and it is only assessed by means of the Observer Scale (found in the computer programme used in the institution), which is a unidimensional scale based solely on the personal perception of the assessor.
We wanted to compare this scale with two others validated for the Portuguese population. These multidimensional scales, based on physiological and behavioural indicators, as indicated in the literature are the Abbey Pain Scale and Pain Assessment in Advanced Dementia (PAINAD). They assess pain at an initial moment and again 45 minutes after an intervention to alleviate pain, so as to determine which is most effective.
The study was conducted on a sample of 76 people, aged 38 to 96 years, the majority of whom were men. Most medical diagnoses were of gastric and/or intestinal diseases (44.8%) and over half of the sample was not submitted to surgery. Almost all of the sample was composed of patients with personal histories of other diseases.
From the results, we concluded that there was a significant relationship between the results of the three scales, most evident in the initial assessment.
The least effective was the Observer Scale, which interestingly is the only one that is applied at this time in the clinical practice mentioned.
The scale which was shown to be more appropriate/effective in the initial assessment was PAINAD and in the assessment after a pain relief intervention, the Abbey Pain Scale.
Since the initial assessment is the most important to invest in better pain management for our patients is the time, prior to any intervention, we concluded that the most appropriate scale for this sample is the Pain Assessment in Advanced Dementia, given the results of the statistical tests used. However, as it is a scale that is recommended for elderly people with dementia, this study confirms that it may also be applied to other age groups, providing the patient has impaired cognition.
Nevertheless, given the small size of our sample, these results cannot be generalized to the population. We therefore suggest further studies in the same field in order to confirm the results with broader samples.
It would also be interesting to apply this scale to patients without impaired cognition in order to determine its effectiveness.
It is in our interest and is our intention to disseminate the results of this work to the institution where the data were collected and to scientific journals and events. This way, we hope, be an inspiration and motivation for best practices based on scientific evidence and for further research, contributing to health gains and to enhancing nursing as a discipline and a science.
References
Batalha, L. (2005). As dificuldades na quantificação exacta da dor. Revista Servir, 53(4), 166-173.
Batalha, L., Duarte, C., Rosário, R., Costa, M., Pereira, V., Morgado, T. (2012). Adaptação cultural e propriedades psicométricas da versão portuguesa da escala Pain Assessment in Advanced Dementia. Revista de Enfermagem Referência, 8, 7-16. Acedido em http://www.scielo.mec.pt/pdf/ref/vserIIIn8/serIIIn8a01.pdf.
Cardoso, A. (2013). Manual de Tratamento da Dor. Lisboa: Lidel – edições técnicas, Lda.
Kazanowski, M. & Laccetti, M. (2005). Dor: fundamentos, abordagem clínica, tratamento. Rio de Janeiro: Editora Guanabara Koogan S.A.
Metzger, C., Muller, A., Schwetta, M. & Walter, C. (2002). Cuidados de enfermagem e Dor. Loures: Lusociência.
Pimenta, C. Santos, E. Chaves, L. Martins, L. & Gutierrez, B. (2001). Controle da dor no pós-operatório. Revista Escola de Enfermagem USP, 35, 2, 180-183. Acedido em http://www.revistas.usp.br/reeusp/article/viewFile/41229/44781.
Rodrigues, A. (2013). Avaliação da Dor ao Doente Oncológico em Cuidados Paliativos Incapaz de Comunicar. Coimbra: Escola Superior de Enfermagem de Coimbra.
Rothrock, J. (2008). Cuidados de Enfermagem ao Paciente Cirúrgico. 13ª edição. Loures: Lusodidacta.
Copyright information
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
About this article
Publication Date
31 July 2016
Article Doi
eBook ISBN
978-1-80296-012-9
Publisher
Future Academy
Volume
13
Print ISBN (optional)
-
Edition Number
1st Edition
Pages
1-462
Subjects
Health psychology, psychology, health systems, health services, ocial issues, teenager, children's health, teenager health
Cite this article as:
Conceição, M., Almeida, V., Duarte, J., Chaves, C., Campos, S., & Nelas, P. (2016). Pain assessment in surgical patients with impaired cognition. In S. Cruz (Ed.), Health & Health Psychology - icH&Hpsy 2016, vol 13. European Proceedings of Social and Behavioural Sciences (pp. 147-157). Future Academy. https://doi.org/10.15405/epsbs.2016.07.02.13