The effect of preoperative vitamin D values on hypocalcemia after total thyroidectomy

Abstract

Objective: Hypocalcemia is a common complication following thyroidectomy. Vitamin D plays a critical role in calcium regulation. This study aimed to investigate the relation between preoperative vitamin D levels and postoperative hypocalcemia.

Material and Methods: We conducted a retrospective analysis of 899 patients who underwent total thyroidectomy at our center between 2015 and 2020 due to multinodular goiter, atypia of undetermined significance, follicular lesions, and follicular neoplasia. Patients were excluded if they had a history of thyroid surgery, Graves’ disease, renal failure, incidental parathyroidectomy, or received calcium or vitamin D supplementation before surgery. The patients were divided into two groups based on their preoperative vitamin D levels: Group 1 (n= 240) with levels <10 ng/mL, and Group 2 (n= 659) with levels ≥10 ng/mL. Demographic characteristics and pre- and postoperative laboratory values were compared between the groups.

Results: The female-to-male ratio was 3.22, with an average vitamin D level of 18.94 ± 13.28 ng/mL. Vitamin D levels were significantly lower in women compared to men (p= 0.001). In Group 1, the rates of asymptomatic and symptomatic postoperative hypocalcemia were 17.1% and 6.7%, respectively; while in Group 2, these rates were 11.2% and 3.2% (p= 0.020). The average preoperative vitamin D level was 14.79 ± 9.4 ng/mL in patients who developed hypocalcemia and 19.12 ± 13.4 ng/mL in those who remained normocalcemic, with this difference being statistically significant (p= 0.026).

Conclusion: In our study, we found that preoperative vitamin D level below 10 ng/mL is associated with increased risk of hypocalcemia following thyroidectomy.

Keywords: Preoperative vitamin D, hypocalcemia, thyroidectomy

Introduction

Thyroidectomy is a common procedure for treating both benign and malignant thyroid diseases. A common complication of total thyroidectomy is hypocalcemia, which can be either asymptomatic or symptomatic. Hypocalcemia can pose significant risks to patient health and prolong hospital stays.

While most instances are temporary, permanent hypocalcemia necessitates lifelong calcium and vitamin D supplementation. Factors contributing to hypocalcemia include patient age, sex, preoperative levels of calcium, vitamin D, and parathyroid hormone (PTH), incidental parathyroidectomy, iatrogenic injury to parathyroid glands, Graves’ disease, reoperation, surgical duration, and the weight of the resected specimen (1,2). Early prediction of hypocalcemia is crucial for timely intervention and reducing morbidity.

In this study, we retrospectively evaluated the demographic characteristics, laboratory results, postoperative hypocalcemia incidence, and intravenous calcium requirements of 899 patients who underwent total thyroidectomy performed by two senior endocrine surgeons. The patients were treated for multinodular goiter, atypia of undetermined significance, follicular lesion of undetermined significance, or follicular neoplasia. The aim was to investigate whether vitamin D levels can predict hypocalcemia.

Material and Methods

We analyzed 899 patients who underwent total thyroidectomy between January 1, 2015, and December 30, 2020, for multinodular goiter, atypia of unknown significance, follicular lesion of unknown significance or follicular neoplasia according to the Bethesda classification. Patients with Graves’ disease, primary hyperparathyroidism, renal insufficiency, central neck dissection, parathyroid reimplantation, or preoperative calcium or vitamin D replacement were excluded from the study. All surgeries were performed using capsular dissection technique and energy-based devices.

We recorded patients’ demographic characteristics [age, sex, body mass index (BMI)], preoperative diagnosis, development of postoperative hypocalcemia, and intravenous calcium requirements by reviewing patient files. We documented preoperative levels of vitamin D, calcium, PTH, phosphorus, albumin, alkaline phosphatase (ALP), and creatinine, as well as postoperative levels of PTH, calcium, phosphorus, and albumin at 12 hours. Length of hospital stay and pathological diagnoses were also recorded and analyzed. Patients were divided into two groups based on their preoperative vitamin D levels: <10 ng/mL (Group 1, n= 240) and ≥10 ng/mL (Group 2, n= 659). Patients with vitamin D levels below 10 ng/mL were considered deficient. Corrected postoperative calcium values were calculated using the formula: Corrected calcium= measured calcium (mg/dL) + 0.8 × [4 - serum albumin (g/dL)].

Symptomatic hypocalcemia was defined by symptoms such as numbness around the mouth, muscle cramps, positive Chvostek or Trousseau signs, and cardiac arrhythmias following total thyroidectomy. Postoperative hypocalcemia was defined as a corrected calcium level below 8 mg/dL, regardless of symptoms. Patients with symptomatic hypocalcemia were treated with intravenous or oral calcium and oral calcitriol. The study was approved by the ethics committee of our institution, and informed consent was obtained from all participants (2018, Number: 4.8.21-153183).

Biochemical Analysis

Vitamin D levels were measured using high-performance liquid chromatography with an auto-analyzer (ZIVAK Technologies, Türkiye). Calcium, PTH, albumin, phosphorus, and ALP levels were measured using spectrophotometric and chemiluminescence methods with an auto-analyzer (Cobas 4000, Roche Diagnostic, Basel, Switzerland). The reference values in our laboratory were= serum calcium 8.2-10.2 mg/dL, vitamin D ≥30 ng/ mL, PTH 15-65 pg/mL, albumin 3.5-5.2 g/dL, phosphorus 2.5-4.5 mg/dL, ALP 40-130 U/L, and creatinine 0.5-0.9 mg/dL.

Statistical Analysis

We used the number cruncher statistical system program for statistical analysis. Descriptive statistics (mean, standard deviation, median, frequency, percentage, minimum, maximum) were employed. Student’s t-test was used for comparing normally distributed quantitative variables between two groups while Mann-Whitney U test was used for non-normally distributed variables. Paired t-test was used for comparing normally distributed variables within the groups, and Wilcoxon signedranks test was used for non-normally distributed variables. Pearson Chi-square test was used for categorical data. A p-value of <0.05 was considered statistically significant.

Results

The study included 899 patients who underwent total thyroidectomy between January 1, 2015, and December 30, 2020. Severe vitamin D deficiency is defined as vitamin D level below 10 ng/mL. Group 1 had 240 patients with vitamin D levels <10 ng/mL, and Group 2 had 659 patients with vitamin D levels ≥10 ng/mL. The mean age, gender distribution, BMI, preoperative diagnosis, and postoperative pathology are detailed in Table 1. A significantly higher rate of low vitamin D levels was observed in females compared to males (p= 0.001) (Table 1).

In Group 1, there was a significant decline of 0.74 ± 0.46 mg/dL in postoperative calcium levels compared to preoperative levels (p= 0.001). Group 2 also experienced a significant decrease of 0.75 ± 0.52 mg/dL in postoperative calcium levels (p= 0.001) (Table 2). In addition, both groups experienced a significant decrease in postoperative albumin levels compared to preoperative levels (Group 1 0.29 ± 0.3 mg/dL, p= 0.001; Group 2 0.32 ± 0.33 units, p= 0.001).

Preoperative phosphorus levels were significantly higher in Group 2 (3.38 ± 0.51 mg/dL) compared to Group 1 (3.28 ± 0.55 mg/dL, p= 0.016). Group 2 had significantly higher preoperative serum creatinine levels compared to Group 1 (0.73 ± 0.17 mg/ dL vs. 0.67 ± 0.15 mg/dL, p= 0.001).

Preoperative and postoperative PTH levels were significantly higher in Group 1 compared to Group 2 (p= 0.001). Group 1 exhibited a significant mean decrease of 21.73 ± 28.71 pg/mL in PTH levels postoperatively (p= 0.001) while Group 2 had a decrease of 16.47 ± 22.42 pg/mL (p= 0.001). The change in postoperative PTH levels was significantly greater in Group 1 compared to Group 2 (p= 0.020) (Table 2).

The incidence of asymptomatic and symptomatic hypocalcemia was lower in Group 2 compared to Group 1 (p= 0.020). There was no significant difference in the need for intravenous calcium supplementation between the groups (p> 0.05) (Table 2). Normocalcemic patients had a significant mean decline of 0.75 ± 0.5 mg/dL in postoperative calcium levels compared to preoperative levels (p= 0.001) while hypocalcemic patients had a decline of 0.93 ± 0.47 mg/dL (p= 0.001). The change in postoperative calcium levels was significantly greater in hypocalcemic cases compared to normocalcemic cases (p= 0.014) (Table 3). There was no significant difference in preoperative PTH values between the groups (p> 0.05). Postoperative PTH levels were significantly higher in normocalcemic cases compared to hypocalcemic cases (p= 0.001). Normocalcemic cases experienced a significant mean PTH decline of 17.19 ± 24.06 pg/mL (p= 0.001) while hypocalcemic cases had a decline of 33.64 ± 26.18 pg/mL (p= 0.001). The change in PTH levels was significantly greater in hypocalcemic cases (p= 0.001). Vitamin D levels were significantly higher in normocalcemic cases compared to hypocalcemic cases (p= 0.026). Postoperative hypocalcemia was observed in 12.79% of 899 patients. Of those 12.79%, 0.01% was permanent and 12.78% was transient hypocalcemia.

Discussion

Our study indicates that low preoperative vitamin D levels may serve as a risk marker for postoperative hypocalcemia. Measuring and managing preoperative vitamin D levels could potentially enhance postoperative outcomes and reduce morbidity and costs.

One of the strengths of our study is that it was conducted on a large and comprehensive patient population. To our knowledge, there are no studies in the literature with a patient series as high as ours that evaluate the effect of vitamin D levels on the risk of hypocalcemia. Another strength is that we excluded patients with other risk factors that could cause hypocalcemia and standardized the impact of different surgical experience. However, a limitation of the study is its retrospective nature.

Hypocalcemia following thyroidectomy is a significant concern due to its impact on patient morbidity, extended follow-up requirements, and increased hospital costs. Permanent hypocalcemia, defined as lasting more than six months, can lead to lifelong calcium and vitamin D supplementation (3). The incidence of temporary hypocalcemia varies widely in the literature, ranging from 0.3% to 49%, while permanent hypocalcemia has been reported 0-13% (4). In our study, postoperative hypocalcemia was observed in 12.79% of 899 patients.

Several factors contribute to postoperative hypocalcemia, including injury or ischemia of the parathyroid glands, incidental parathyroidectomy, parathyroid reimplantation, female sex, Graves’ disease, the surgeon’s experience and technique, preoperative calcium and vitamin D levels, PTH levels, reoperation, and extensive neck dissection (1,5,6). A prospective study indicated that surgical technique is a major determinant of postoperative hypocalcemia (7). In our study, all thyroidectomies were performed by two experienced endocrine surgeons using a consistent technique.

Despite numerous studies investigating risk factors for postoperative hypocalcemia, the role of preoperative vitamin D deficiency remains controversial. Some studies support the link between vitamin D deficiency and hypocalcemia, while others do not (8-11). Vitamin D deficiency can exacerbate postoperative hypocalcemia because, in the presence of vitamin D deficiency, PTH compensates by increasing calcium reabsorption from bones and kidneys to maintain normocalcemia (12). However, when transient hypoparathyroidism occurs post-surgery, patients with vitamin D deficiency are more prone to hypocalcemia due to the loss of this compensatory mechanism (13). A study found that patients with preoperative vitamin D deficiency had a higher risk of hypocalcemia and elevated postoperative PTH levels compared to those with normal vitamin D levels (14). Conversely, another study suggested that preoperative PTH measurement could be misleading due to secondary hyperparathyroidism in patients with vitamin D deficiency (15). Vitamin D deficiency is a prevalent issue in Türkiye (16,17). In our study, a cut-off value of 10 ng/mL for preoperative vitamin D was used. Severe vitamin D deficiency is encountered below this cut-off value. Using a lower cut-off value allows to examine the impact of severe deficiency on postoperative outcomes, providing insights into whether very low vitamin D levels are particularly predictive of hypocalcemia. Our study showed that patients with vitamin D levels <10 ng/mL had higher rates of biochemical and symptomatic hypocalcemia (17.1% and 6.7%, respectively) compared to those with levels ≥10 ng/mL (11.2% and 3.2%, respectively) (p= 0.020). This result demonstrates a significant effect of preoperative vitamin D levels on postoperative hypocalcemia. Similarly, in a prospective study of 166 patients with three groups based on their vitamin D levels: 20 ng/mL, the study found postoperative hypocalcemia rates of 32%, 24%, and 13%, respectively, highlighting a significant association between vitamin D levels and postoperative hypocalcemia (14). In another study, vitamin D cut-off of <30 ng/mL significantly increased the risk of postoperative hypocalcemia (18). In our study, mean vitamin D level was 14.79 ± 9.4 ng/mL in hypocalcemic patients and 19.12 ± 13.4 ng/mL in normocalcemic patients, with a significant difference between these groups (p= 0.026; p< 0.05).

Although many studies support the association between low preoperative vitamin D levels and increased risk of postoperative hypocalcemia, some studies indicate no significant correlation between preoperative vitamin D levels and postoperative hypocalcemia (19-22). For example, a retrospective study has found no predictive value of preoperative vitamin D levels for postoperative hypocalcemia in benign thyroid conditions (23). One study has even suggested that low preoperative vitamin D levels might have a protective effect against hypocalcemia by inducing parathyroid hypertrophy and increased PTH secretion (24). Another study has concluded that PTH levels, rather than vitamin D levels, are the most accurate predictor of postoperative symptomatic hypocalcemia (10). Our study observed significant differences in postoperative calcium levels and hypocalcemia incidence (p= 0.017, p= 0.020), aligning with the literature.

A prospective study noted a significant decrease in pre- and postoperative PTH levels among those with vitamin D deficiency, suggesting that postoperative PTH levels were inadequate for predicting hypocalcemia (25). In our study, significant changes in preoperative and postoperative PTH levels were observed in vitamin D deficient patients (Group 1) compared to those with sufficient vitamin D levels (Group 2) (p= 0.001, p= 0.020). Postoperative PTH levels were lower in hypocalcemic patients, with a greater change compared to normocalcemic patients (p= 0.001). No significant relationship was found between preoperative PTH levels and hypocalcemia (p= 0.230). Other studies also demonstrate the association between postoperative PTH levels and hypocalcemia risk (25-27).

Our study revealed a significant sex difference, with a higher rate of vitamin D deficiency in women compared to men (p= 0.001). The incidence of hypocalcemia was also higher in females (14.6% vs. 7%, p= 0.002). Consistent with our findings, some studies show lower preoperative vitamin D levels and higher hypocalcemia rates in women (28). Additionally, female gender is often associated with increased risk of postoperative hypocalcemia (29,30).

Additionally, our study noted significant sex differences, with higher rates of vitamin D deficiency and postoperative hypocalcemia in women (p= 0.001; p= 0.002). This finding is consistent with some literature that reports lower preoperative vitamin D levels and higher hypocalcemia rates in women (28-30). Furthermore, we found that patients with postoperative hypocalcemia were significantly younger (p= 0.003) though no significant correlation between vitamin D levels and age was observed (p= 0.16).

We found that patients with postoperative hypocalcemia were significantly younger than those with normal calcium levels (p= 0.003) although no significant correlation was found between vitamin D levels and age (p= 0.16). One study reported that older patients had a higher mean age and a greater risk of postoperative hypocalcemia (28).

Although our study encompassed a wide patient population, there are conflicting findings in the literature regarding this topic. Further studies are needed to confirm and support our findings.

Conclusion

According to our data, preoperative vitamin D level below 10 ng/mL is associated with postoperative hypocalcemia.

Cite this article as: Qanbar MM, Soylu S, Teksöz E. The effect of preoperative vitamin D values on hypocalcemia after total thyroidectomy. Turk J Surg 2024; 40 (4): 296-302.

This study was approved by İstanbul University-Cerrahpaşa Cerrahpaşa Faculty of Medicine Clinical Research Ethics Committee (Decision no: 153183, Date: 04.08.2021).

Externally peer-reviewed.

Concept - All of authors; Design - All of authors; Supervision - ST, SS; Materials - SS, MMQ; Data Collection and/or Processing - MMQ, SS; Analysis and/or Interpretation - All of authors; Literature Search - MMQ, SS; Writing Manuscript - All of authors; Critical Reviews - All of authors.

The authors have no conflicts of interest to declare.

The authors declared that this study has received no financial support.

References

1. Edafe O, Antakia R, Laskar N, Uttley L, Balasubramanian SP. Systematic review and meta-analysis of predictors of post-thyroidectomy hypocalcaemia. Br J Surg 2014; 101(4): 307-20. https://doi. org/10.1002/bjs.9537
2. Hallgrimsson P, Nordenström E, Bergenfelz A, Almquist M. Hypocalcaemia after total thyroidectomy for Graves’ disease and for benign atoxic multinodular goitre. Langenbeck’s Arch Surg 2012; 397(7): 1133-7. https://doi.org/10.1007/s00423-012-0981-1
3. Qin Y, Sun W, Wang Z, Dong W, He L, Zhang T, et al. A meta-analysis of risk factors for transient and permanent hypocalcemia after total thyroidectomy. Front Oncol 2021; 10.
4. Lang BH-H, Wong KP, Cheung CY, Fong YK, Chan DK-K, Hung GK-Y. Does preoperative 25-hydroxyvitamin d status significantly affect the calcium kinetics after total thyroidectomy? World J Surg 2013; 37(7): 1592-8. https://doi.org/10.1007/s00268-013-2015-8
5. Thomusch O, Machens A, Sekulla C, Ukkat J, Lippert H, Gastinger I, et al. Multivariate analysis of risk factors for postoperative complications in benign goiter surgery: Prospective multicenter study in Germany. World J Surg 2000; 24(11): 1335-41. https://doi.org/10.1007/ s002680010221
6. Gambardella C, Patrone R, Di Capua F, Offi C, Mauriello C, Clarizia G, et al. The role of prophylactic central compartment lymph node dissection in elderly patients with differentiated thyroid cancer: A multicentric study. BMC Surg 2019; 18(S1).
7. Erbil Y, Ozbey NC, Sari S, Unalp HR, Agcaoglu O, Ersöz F, et al. Determinants of postoperative hypocalcemia in vitamin D-deficient Graves’ patients after total thyroidectomy. Am J Surg 2011; 201(5): 685-91. https://doi.org/10.1016/j.amjsurg.2010.04.030
8. Al-Khatib T, Althubaiti AM, Althubaiti A, Mosli HH, Alwasiah RO, Badawood LM. Severe vitamin D deficiency: A significant predictor of early hypocalcemia after total thyroidectomy. Otolaryngol Head Neck Surg 2015; 152(3): 424-31. https://doi.org/10.1177/0194599814561209
9. Tripathi M, Karwasra R, Parshad S. Effect of preoperative vitamin D deficiency on postoperative hypocalcemia after thyroid surgery. Thyroid Res 2014; 7(1): 8. https://doi.org/10.1186/1756-6614-7-8
10. Deffain A, Scipioni F, De Rienzo B, Allal S, Castagnet M, Kraimps JL, et al. Preoperative vitamin D levels do not relate with the risk of hypocalcemia following total thyroidectomy. A cohort study. Minerva Chir 2019; 74(1). https://doi.org/10.23736/S0026-4733.19.08057-X
11. Salinger EM, Moore JT. Perioperative indicators of hypocalcemia in total thyroidectomy: The role of vitamin D and parathyroid hormone. Am J Surg 2013; 206(6): 876-82.
12. Nygren P, Larsson R, Johansson H, Ljunghall S, Rastad J. 1,25(OH)2 D3 inhibits hormone secretion and proliferation but not functional dedifferentiation of cultured bovine parathyroid cells. Calcif Tissue Int 1988; 43(4): 213-8. https://doi.org/10.1007/BF02555137
13. Bove A, Dei Rocini C, Di Renzo RM, Farrukh M, Palone G, Chiarini S, et al. Vitamin D deficiency as a predictive factor of transient hypocalcemia after total thyroidectomy. Int J Endocrinol 2020; 2020: 1-6.
14. Kirkby-Bott J, Markogiannakis H, Skandarajah A, Cowan M, Fleming B, Palazzo F. Preoperative vitamin D deficiency predicts postoperative hypocalcemia after total thyroidectomy. World J Surg 2011; 35(2): 324-30. https://doi.org/10.1007/s00268-010-0872-y
15. Unsal IO, Calapkulu M, Sencar ME, Hepsen S, Sakiz D, Ozbek M, et al. Preoperative vitamin D levels as a predictor of transient hypocalcemia and hypoparathyroidism after parathyroidectomy. Sci Rep 2020; 10(1): 9895.
16. Satman I, Ozbey N, Boztepe H, Kalaca S, Omer B, Tanakol R, et al. Prevalence and correlates of vitamin D deficiency in Turkish adults. Endocr Abstr 2013; 25.
17. Atli T, Gullu S, Uysal AR, Erdogan G. The prevalence of Vitamin D deficiency and effects of ultraviolet light on Vitamin D levels in elderly Turkish population. Arch Gerontol Geriatr 2005; 40(1): 53-60.
18. Rubin SJ, Park JH, Pearce EN, Holick MF, McAneny D, Noordzij JP. Vitamin D status as a predictor of postoperative hypocalcemia after thyroidectomy. Otolaryngol Neck Surg 2020; 163(3): 501-7.
19. de Carvalho GB, Giraldo LR, Lira RB, Macambira IBM, Tapia MA, Kohler HF, et al. Preoperative vitamin D deficiency is a risk factor for postoperative hypocalcemia in patients undergoing total thyroidectomy: Retrospective cohort study. Sao Paulo Med J 2019; 137(3): 241- 7. https://doi.org/10.1590/1516-3180.2018.0336140319
20. Alkhalili E, Ehrhart MD, Ayoubieh H, Burge MR. Does pre-operative vitamin D deficiency predict postoperative hypocalcemia after thyroidectomy? Endocr Pract 2017; 23(1): 5-9. https://doi. org/10.1177/1078155217691227
21. Vibhatavata P, Pisarnturakit P, Boonsripitayanon M, Pithuksurachai P, Plengvidhya N, Sirinvaravong S. Effect of preoperative vitamin D deficiency on hypocalcemia in patients with acute hypoparathyroidism after thyroidectomy. Int J Endocrinol 2020; 2020: 1-9.
22. Dağlar G, Kılıç M, Çelik C, Yüksel C, Terzioğlu SG, Özden S, et al. Is there a relationship between vitamin D status and hypocalcemia after total thyroidectomy? Acta Endocrinol (Copenh) 2016; 12(3): 291-6. https://doi.org/10.4183/aeb.2016.291
23. Erlem M, Klopp-Dutote N, Biet-Hornstein A, Strunski V, Page C. Impact of pre-operative serum 25-hydroxyvitamin D on post-operative serum calcium in patients undergoing total thyroidectomy for benign goitre: Retrospective study of 246 patients. J Laryngol Otol 2017; 131(10): 925-9. https://doi.org/10.1017/S0022215117001797
24. Chia SH, Weisman RA, Tieu D, Kelly C, Dillmann WH, Orloff LA. Prospective study of perioperative factors predicting hypocalcemia after thyroid and parathyroid surgery. Arch Otolaryngol Neck Surg 2006; 132(1): 41. https://doi.org/10.1001/archotol.132.1.41
25. Lin Y, Ross HL, Raeburn CD, DeWitt PE, Albuja-Cruz M, Jones EL, et al. Vitamin D deficiency does not increase the rate of postoperative hypocalcemia after thyroidectomy. Am J Surg 2012; 204(6): 888-94. https://doi.org/10.1016/j.amjsurg.2012.10.001
26. Griffin TP, Murphy MS, Sheahan P. Vitamin D and risk of postoperative hypocalcemia after total thyroidectomy. JAMA Otolaryngol Neck Surg 2014; 140(4): 346-51.
27. Nhan C, Dolev Y, Mijovic T, Rivera JA, Kallai-Sanfaçon MA, Mlynarek AM, et al. Vitamin D deficiency and the risk of hypocalcemia following total thyroidectomy. J Otolaryngol Head Neck Surg 2012; 41(6): 401-6.
28. Pradeep PV, Ramalingam K. Postoperative PTH Measurement is not a reliable predictor for hypocalcemia after total thyroidectomy in vitamin D deficiency: Prospective study of 203 cases. World J Surg 2014;38(3):564-7. https://doi.org/10.1007/s00268-013-2350-9
29. Vanderlei FAB, Vieira JGH, Hojaij FC, Cervantes O, Kunii IS, Ohe MN, et al. Parathyroid hormone: An early predictor of symptomatic hypocalcemia after total thyroidectomy. Arq Bras Endocrinol Metabol 2012;56(3):168-72. https://doi.org/10.1590/S0004- 27302012000300003
30. Chow TL, Choi CY, Chiu ANK. Postoperative PTH monitoring of hypocalcemia expedites discharge after thyroidectomy. Am J Otolaryngol 2014;35(6):736-40. https://doi.org/10.1016/j.amjoto.2014.07.006